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FOREWORD 


One  of  the  missions  of  the  Army  Research  Institute  for  the 
Behavioral  and  Social  Sciences  (ARI )  is  to  conduct  research  and 
development  in  individual  training  methods.  The  Army  has 
established  the  Training  Technology  Field  Activities  ( TTFA )  to 
determine  how  and  where  training  technology  can  be  used  to 
provide  cost-effective  military  training  .  TTFAs  are  located  at 
several  Army  schools  and  are  organized  as  partnerships  consisting 
of  ARI,  the  Training  and  Doctrine  Command  ( TRADOC ) ,  and  the 
particular  school.  This  report  is  part  of  ARI's  support  of  the 
TTFAs . 


The  task  to  develop  a  theo 
TTFAs  to  systematically  compa 
combined  cost  and  effectiveness 
Consortium  of  Universities  o 
and  monitored  by  ARI.  The  cont 
a  series  let  by  ARI  intended 
to  bear  on  the  Army's  personnel 
Defense  Department's  effort  to 
base . 


retical  model  which  would  enable 
re  alternative  ways  of  training  on 
variables  was  contracted  to  the 
f  the  Washington  Metropolitan  Area 
ract  with  the  Consortium  is  one  of 
to  bring  the  expertise  of  academia 
research  and  also  support  the 
strengthen  the  university  research 


The  products  of  this  contract  are  four  reports,  two  which 
review  pertinent  literature  and  two  which  treat  the  development 
of  the  theoretical  model.  Anyone  concerned  with  cost  and 
training  effectiveness  analysis  or  the  requirements  of  model 
development  will  find  these  reports  comprehensive  and  thorough. 
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A  REVIEW  OF  MODELS  OF  COST  AND  TRAINING  EFFECTIVENESS  ANALYSIS, 
VOLUME  II:  COST  ANALYSIS 


EXECUTIVE  SUMMARY 


Requirement: 

To  identify  cost  and  training  effectiveness  analysis  (CTEA) 
models  which  might  be  adapted  to  the  needs  of  the  Army  for  im¬ 
proving  its  training  development  system,  particularly  in  areas 
involved  with  use  of  new  training  technology.  The  focus  of  this 
report  is  on  models  that  deal  with  training  costs.  The  companion 
report,  Volume  I:  Training  Effectiveness  Analysis,  examines  models 
concerned  with  measures  of  training  effectiveness. 

Procedure : 

A  body  of  literature  assev  ’  led  by  the  Army  Research  Institute 
and  literature  already  in  possession  of  the  authors  was  searched. 
This  literature  enabled  the  devlopment  of  a  keyword  search  of 
computerized  data  bases  which  included  the  Defense  Technical 
Information  Center  (DTIC) ,  the  National  Technical  Information 
Service  (NTIS) ,  the  Educational  Resources  Information  Center 
(ERIC) ,  and  PsychScan.  References  were  also  obtained  through 
contact  with  several  training  and  cost  experts  and  major  consulting 
firms . 

Findings: 

The  report  provides  the  economic  background  and  framework 
needed  to  understand  and  evaluate  CTEA  models.  Seventeen  models  of 
potential  applicability  to  training  technology  decisions  are 
reviewed  and  analyzed.  The  major  impediment  to  utilization  of  most 
models  is  seen  as  resulting  from  their  inability  to  treat  the  costs 
of  training  development.  Models  which  generate  and  process 
estimates  based  on  experience  with  similar  forms  of  training 
technology  are  considered  useful  tools. 

Utilization  of  Findings: 

This  literature  review  will  be  valuable  to  researchers  who 
need  a  grounding  in  CTEA  models.  It  will  also  be  of  interest  to 
decision-makers  who  wish  to  know  something  of  the  internal  workings 
of  a  model  and  where  the  trade-offs  reside.  The  authors  are  using 
the  contents  of  this  report  to  develop  a  CTEA  model  called  Training 
and  Cost  Iterative  Technique  (TECIT)  which  will  be  described  in 
subsequent  publications. 
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COST  AND  TRAINING  EFFECTIVENESS  ANALYSIS:  A  REVIEW 
OF  LITERATURE  ON  COST  MODELS 

I.  INTRODUCTION 

THE  PROBLEM 

The  fielding  of  new  weapon  systems  and  technologies  in  the 
nation's  defense  requires  the  training  of  those  who  will  use 
them.  The  effectiveness  of  these  systems  and  the  success  of  new 
technologies  depends  to  a  large  degree  on  investments  in  human 
capital  alongside  those  in  physical  capital.  The  human  element 
plays  an  important  role  in  even  the  most  capital  intensive  system 
affecting  the  capacity  of  the  system  to  carry  out  its  mission 
successfully.  The  design  of  a  weapon  system  or  technology  can 
directly  affect  the  size  of  investment  in  training  required  as 
well  as  the  effectiveness  of  both  when  fielded.  Overly  complex 
technologies  may  introduce  additional  cost  in  training  and  diminish 
the  effectiveness  of  these  technologies  when  fielded.  It  is  even 
possible  for  poorly  designed  systems  to  fail  altogether  due  to 
the  human  element. 

The  Army,  like  other  Services,  has  been  aware  of  this 
relationship  between  man  and  systems.  Over  the  past  two  decades. 


the  Army  has  invested  heavily  in  research  to  develop  models  of 
this  relationship  allowing  it  to  predict  cost  and  performance  of 
new  weapon  systems  and  technologies  as  they  relate  to  training 
and  the  human  element.  Cost  and  Training  Effectiveness  Analysis 
(CTEA)  models  developed  for  this  purpose  have  been  used  in  a 
wide  variety  of  settings  by  the  Army.  Two  problems  have  plagued 
the  use  of  CTEA  models.  The  first  is  the  search  for  generic 
models  which  can  be  adapted  to  a  full  range  of  systems  and 
technologies.  Development  to  date  has  been  concentrated  on 
specific  models  intended  for  a  limited  range  of  applications. 

The  second  is  the  search  for  models  which  can  be  used  in  the 
early  stages  of  development  of  a  new  system  or  technology  to 
predict  the  success  of  alternative  designs. 

The  earlier  one  can  anticipate  the  cost  and  effectiveness 
of  a  design,  the  easier  it  is  to  adjust  the  design  to  address 
deficiencies  leading  to  excessive  cost  and  impediments  to  its 
effectiveness.  Every  weapon  system  and  technology  passes  through 
a  development  life-cycle  beginning  with  a  conceptual  stage  and 
continuing  through  development,  procurement,  operations  and 
maintenance  stages.  Design  changes  at  later  stages  of  the  life- 
cycle,  frequently  requiring  expensive  retooling  and  retrofitting, 
can  be  avoided  through  early  assessment  of  the  impact  a  design 
will  have  on  the  human  element  of  its  operation.  CTEA  models, 
however,  have  encountered  problems  in  the  prediction  of  cost  and 
training  effectiveness  during  early  stages  of  the  life-cycle. 


The  problem  is  in  part  linked  to  the  absence  of  historical  data 
for  evaluation  of  these  issues,  but  also  to  appropriate  prediction 
methodologies . 

Recently,  the  Army  established  a  new  unit  within  its  Training 
and  Doctrine  Command  (TRAD OC)  charged  with  the  improvement  of 
training,  particularly  that  involved  with  the  use  of  new  training 
technologies .  The  Army  Training  Technology  Field  Activities 
(TTFA's)  are  a  joint  effort  of  TRADOC ,  the  Army  Research  Institute, 
and  selected  Army  Schools  to  investigate  the  cost  and  effectiveness 
of  new  training  technologies.  Among  these  technologies  is  that 
of  computer-based  instruction.  The  TTFA's  will  employ  CTEA 
models  in  their  investigation. 

OBJECTIVES 

To  support  this  effort  and  the  broader  task  of  improving 
the  Army's  ability  to  predict  the  cost  and  effectiveness  of  new 
weapon  systems  and  technologies  early  in  their  life-cycle,  the 
Consortium  of  Washington  Area  Universities,  represented  by  the 
University  of  the  District  of  Columbia  and  the  George  Washington 
University,  were  assigned  the  task  of  developing  a  CTEA  model 
oriented  toward  the  needs  of  TTFA's,  as  well  as  other  interested 
Army  and  Department  of  Defense  organizations. 

This  report,  prepared  by  the  George  Washington  University, 
will  focus  on  CTEA  cost  literature.  A  companion  report,  prepared 
by  the  University  of  the  District  of  Columbia,  will  address  the 


CTEA  training  effectiveness  literature.  The  objective  is  to 
identify  models  which  might  be  adapted  to  the  needs  of  TTFA's 
and  to  extract  the  lessons  to  be  learned  from  the  use  of  these 
models  in  other  settings  that  could  enhance  their  contribution 
to  the  TTFA's.  This  review  will  be  followed  in  a  second  phase 
of  study  by  the  development  of  a  CTEA  model  along  with  a  validation 
plan  for  use  by  the  TTFA's. 

This  review  of  CTEA  cost  literature  builds  on  several  earlier 
reviews  of  these  models  including  those  of  Matlick  et  al  (1984), 
Rosen  et  al  (1981),  and  Knerr  et  al  (1984).  The  report  will 
emphasize  new  developments  and  issues  in  the  literature  which 
are  determined  to  be  relevant  to  the  development  of  a  predictive 
CTEA  model  serving  the  needs  of  TTFA's. 

APPROACH  TO  THE  STUDY 

The  literature  search  began  with  a  body  of  CTEA  literature 
assembled  by  the  Army  Research  Institute  (ARI)  and  literature 
already  in  the  possession  of  the  authors.  A  review  of  this 
literature  and  its  references  led  to  the  identification  of  other 
sources  which  appeared  useful  to  our  efforts.  This  literature 
also  served  as  the  means  for  developing  a  keyword  search  of 
computerized  data  bases  spanning  civilian  and  military  literature. 
The  data  bases  searched  included  the  Defense  Technical  Information 
Center  (DTIC),  the  National  Technical  Information  Service  (NTIS), 
the  Educational  Resources  Information  Center  (ERIC),  and  PsychScan. 


The  search  also  included  references  obtained  through  contact 
with  several  training  and  cost  experts  and  major  consulting 
firms . 

Using  keywords  focusing  on  training  and  cost  effectiveness, 
the  DTIC  search  yielded  some  350  abstracts.  ERIC  and  PsychScan 
identified  approximately  250  references,  while  NTIS  produced  an 
additional  300  references.  An  examination  of  these  abstracts 
for  the  relevance  to  the  task  at  hand  led  to  a  narrowing  of  the 
number  of  studies  actually  reviewed.  A  bibliography  of  the 
literature  reviewed  is  included  with  this  report. 

ORGANIZATION  OF  THE  REPORT 

The  report  is  organized  in  four  sections.  Section  II, 
which  follows,  provides  a  lexicon  of  the  methods  and  concepts 
used  in  developing  cost  estimates  for  CTEA.  The  section  provides 
a  framework  for  reviewing  CTEA  cost  literature  in  Section  III. 

The  reader  who  is  familiar  with  these  methods  and  concepts  will 
want  to  proceed  to  Section  III.  In  Section  III,  17  CTEA  models 
and  guidebooks  are  examined.  The  models  and  guidebooks  are 
described  and  compared  with  respect  to  selected  features.  The 
analysis  examines  the  purpose  for  which  each  model  was  developed 
and  appraises  its  strengths  and  weaknesses  for  the  purpose  at 
hand.  Section  IV  concludes  with  a  summary  of  the  lessons  learned 
from  the  literature  for  the  development  of  a  CTEA  cost  model. 


Rational  choice  among  alternative  instructional  delivery 
systems  requires  the  consideration  of  cost  alongside  training 
effectiveness.  It  makes  little  sense  to  select  the  most  effective 
system  to  satisfy  a  particular  training  requirement  without 
regard  to  differences  in  cost.  At  the  same  time,  selection  of 
the  least-cost  alternative  in  the  interest  of  cost  savings  or 
cost  avoidance  may  result  in  an  unacceptably  low  level  of  training 
effectiveness.  Cost  and  training  effectiveness,  as  such,  must 
be  considered  jointly. 

As  suggested  by  Orlansky  (1985),  new  training  technology 
can  be  recommended  for  adoption  when  it  is  more  effective  than 
current  technology  and  costs  the  same  or  less ;  or  when  it  has 
about  the  same  effectiveness  but  costs  less  than  current  technology 
Where  a  trade-off  between  costs  and  effectiveness  is  present, 
the  choice  of  an  alternative  might  be  preferable  if  much  higher 
levels  of  effectiveness  can  be  achieved  with  only  slightly  higher 
expenditures.  By  the  same  token,  small  reductions  in  effectiveness 
might  be  accepted  when  accompanied  by  large  reductions  in  cost. 

The  development  of  a  CTEA  model  formalizes  the  process  of 
choosing  among  alternative  training  technologies  by  comparing 
the  cost  and  training  effectiveness  of  these  systems.  Within 
this  framework,  a  cost  model  is  employed  to  value  the  resources 


engaged  in  each  system.  The  model  requires  the  alternative  systems 
be  identified  and  their  associated  resources  specified.  It  then 
provides  an  algorithm  for  determining  the  cost  of  these  systems. 

The  model,  as  such,  provides  rules  for  the  conduct  of  the  cost 
analysis  and  assures  a  systematic  result  for  comparison.  The 
model,  as  discussed  below,  may  serve  more  than  one  objective 
which  will  influence  its  specification. 

The  resources  engaged  in  alternative  training  technologies 
conventionally  include  facilities,  equipment,  instructional 
material,  supplies,  and  personnel.  The  expenditure  of  these 
resources  is  calculated  for  each  alternative  over  the  life-cycle 
of  a  weapon  system  or  technology.  For  training  associated  with 
a  new  weapon  system,  it  is  important  that  the  CTEA  and  cost 
estimation  take  place  as  early  in  the  system  life-cycle  as  possible. 
The  prediction  of  training  cost  as  affected  by  the  design  of  a 
weapon  system  is  an  important  management  tool  for  achieving 
system  objectives.  Changes  in  system  design  to  control  these 
costs  and  meet  objectives  can  be  undertaken  at  a  lower  cost  in 
the  early  stages  of  a  system  life-cycle  than  in  later  stages. 


Cost  Objectives 

The  cost  to  be  measured  at  each  stage  of  a  system  life- 
cycle  will  vary  with  the  objectives  of  the  analysis.  Braby,et  al 
(1975)  point  out  that  when  the  objective  of  the  analysis  is  to 
select  the  least-cost  alternative  from  among  a  specified  set  of 
training  technologies,  all  of  which  are  capable  of  meeting  the 


training  objectives,  then  the  resources  common  to  all  alternatives 
can  be  factored  out  and  ignored  in  the  analysis.  However,  when 
the  objective  is  to  determine  the  absolute  life-cycle  cost  of  a 
system,  then  all  resources  must  be  included  and  valued  at  their 
opportunity  cost. 

The  use  of  a  cost  model  for  CTEA  requires  that  the  objectives 
of  the  analysis  be  clearly  specified.  In  most  instances,  the 
decision  to  provide  training  has  already  been  made.  The  issue 
is  how  to  provide  the  training  in  an  effective  and  efficient 
manner.  In  this  case,  the  factoring  out  of  common  resources 
among  the  alternatives  and  valuing  the  remainder  is  appropriate. 
However,  if  the  decision  is  whether  or  not  to  offer  training, 
then  the  cost  of  all  resources  over  the  life-cycle  of  a  system 
should  be  compared  with  the  benefits  of  the  training.  If  the 
decision  is  to  be  based  on  a  system’s  affordability,  then  budgetary 
cost  may  enter  the  picture. 

Because  of  the  different  cost  estimates  resulting  from 
these  approaches,  it  is  important  that  the  objectives  served  by 
the  cost  model  be  clearly  specified.  Typically,  the  analyst 
will  be  confronted  with  a  set  of  alternative  training  technologies, 
all  of  which  have  been  determined  acceptable.  The  problem  in 
this  instance  is  one  of  cost-minimization.  Common  resources 
among  the  alternatives  can  be  factored  out  and  the  remainder 
valued.  The  output  of  the  model  may  be  expressed  in  forms  related 
to  unit  cost,  such  as  the  average  cost  per  graduate  or  the  average 


cost  per  student  position,  or  in  terms  of  time  and  flows,  such 
as  the  annual  cost  of  operating  a  school  for  a  specified  number 
of  students. 

Level  of  Cost  Aggregation 

The  use  of  a  cost  model  for  CTEA  also  requires  that  the 
level  of  aggregation  be  specified  for  the  analysis.  An  analyst 
may  choose  to  partition  the  training  into  tasks  and  to  produce 
cost  estimates  for  each  task  and  then  sum  these  estimates  to 
arrive  at  a  total  program  cost.  Alternatively,  total  program 
cost  estimates  may  be  produced  directly.  The  latter  may  prove 
difficult,  especially  during  the  conceptual  stage  of  a  weapon 
system's  development  where  the  absence  of  hard  information  may 
introduce  substantial  uncertainty  into  the  cost  estimates.  The 
identification  of  a  task  or  tasks  on  fielded  weapon  systems 
whose  characteristics  are  comparable  to  a  task  on  the  system  to 
be  developed  may  open  the  way  to  information  reducing  the  level 
of  uncertainty  in  the  cost  estimates  at  this  key  stage  of  the 
weapon  system's  life-cycle. 

Generic  Cost  Models 

The  search  for  a  generic  CTEA  model,  and  by  inference,  a 
generic  cost  model  is  a  theme  found  throughout  the  CTEA  literature. 
The  concept  of  a  model  for  all  seasons  and  circumstances  is 
intuitively  appealing,  although  difficult  to  obtain.  The  objective 
is  to  produce  a  model  that  can  be  used  in  cost  estimation  for 


training  technologies  varying  from  classroom  lectures,  to  computer- 
based  instruction,  to  simulators  and  other  training  devices. 
Moreover,  the  model  should  be  flexible  enough  to  produce  cost 
estimates  at  any  stage  of  a  weapon  system's  life-cycle. 

The  goal  of  a  systematic  method  for  cost  and  training 
effectiveness  evaluation  suited  to  any  situation  is  achievable 
at  a  cost  in  the  level  of  user  sophistication  required  to  implement 
the  model.  Generic  models  by  their  very  nature  operate  at  a 
much  higher  level  of  abstraction  than  specific  models.  The 
enumeration  of  resources  engaged  in  computer-based  instruction, 
for  example,  would  be  more  fully  specified  in  a  specific  cost 


model  than  in  a  generic  model.  The  latter  would  demand  a  higher 
level  of  user  sophistication  than  the  former.  That  is,  the  user 
must  have  a  higher  order  of  working  familiarity  with  the  training 
technology  in  order  to  implement  a  generic  cost  model.  This 
tradeoff  should  be  considered  in  the  development  of  a  CTEA  model 
for  the  benefit  of  those  expected  to  conduct  the  analysis. 

Location  and  Level  of  Training 

Training  for  any  weapon  system  or  occupational  specialty 
may  take  place  in  a  variety  of  locations  and  levels  which  can 
produce  special  problems  for  the  cost  analyst.  In  many  situations 
training  will  occur  initially  in  a  school  setting  followed  by 
assignment  to  a  unit  where  additional  training  will  take  place. 
Some  of  the  training  in  each  setting  may  be  for  the  individual 


and  some  for  the  individual  as  part  of  a  unit.  The  recognition 
of  this  is  important  to  the  proper  specification  of  the  cost 
model.  A  task  oriented  model,  for  example,  may  treat  the  training 
setting  as  a  variable  to  determine  the  cost  effectiveness  of 
task  training  in  alternative  settings.  Most  cost  models  appear 
to  focus  on  individual  training  in  an  institutional  setting. 

Admonitions  and  Model  Limitations 


In  all  choices,  the  analyst  must  keep  in  mind  the  applications 
and  objectives  of  the  CTEA  exercise.  These  elements  of  the 
exercise  must  be  clearly  stated  and  understood  by  all  to  ensure 
the  model  is  properly  specified  and  to  provide  a  means  for  the 
critical  appraisal  of  its  result.  The  analyst  should  also 
acknowledge  the  limitations  of  a  cost  model  that  merely  sums  the 
value  of  resources  employed  in  a  training  program  without  relating 
these  resources  to  a  production  function.  The  latter  is  necessary 
to  the  determination  of  optimal  combinations  of  resources  in 
achieving  different  levels  of  training  output. 

The  conduct  of  cost  modeling  for  CTEA  involves  both  art  and 
science.  The  economic  concepts  employed,  described  below,  lend 
rigor  to  the  process.  The  operationalizing  of  these  concepts, 
however,  is  sometimes  an  art  form.  The  choice  of  a  discount 
rate,  for  example,  or  the  measurement  of  an  indirect  cost  freguently 
requires  the  analysts's  judgment.  In  all  instances,  these  judgments 
should  be  clearly  identified.  In  general,  the  practice  of 


sensitivity  analysis  is  appropriate  in  cost  modeling  as  in  other 
exercises  to  determine  the  sensitivity  of  results  to  the  analyst's 
assumptions . 


ECONOMIC  CONCEPTS  IN  COST  MODELING 


Opportunity  Cost  versus  Accounting  Cost 

CTEA  cost  models  place  a  value  on  the  resources  used  by 
alternative  training  technologies.  These  resources  are  valued 
in  terms  of  their  opportunity  cost,  which  may  differ  from  their 
value  in  terms  of  accounting  costs.  The  opportunity  cost  of  a 
resource  used  in  training  is  measured  by  valuing  the  goods  or 
services  the  resource  could  be  used  to  produce  in  its  best 
alternative  use.  A  classroom  instructor  training  individuals  in 
the  techniques  of  repairing  communications  satellites,  for  example, 
might  alternatively  be  employed  full-time  in  the  repair  of  these 
satellites.  The  opportunity  cost  of  the  time  spent  in  the  classroom 
as  such,  is  measured  by  the  value  of  the  foregone  repair  services. 

Accounting  costs  will  include  direct  payments  for  resources 
used,  but  unlike  opportunity  costs,  will  omit  implicit  payments. 

The  accountant,  for  example,  will  treat  the  depreciation  of  a 
classroom  building  used  for  training  as  a  direct  cost  in  an 
accounting  period,  but  will  ignore  the  foregone  income  potential 
of  the  undepreciated  portion  of  the  building.  The  book  value  ai 


undepreciated  portion  of  the  building,  if  it  could  be  liquidated 
and  the  proceeds  invested,  would  produce  a  stream  of  income 
during  the  accounting  period.  This  foregone  income  is  an  implicit 
cost  of  keeping  the  building  in  service  for  future  use  and  is 
not  included  by  the  accountant  in  the  cost  of  training,  but 
would  be  included  in  the  opportunity  cost. 

Time  and  market  conditions  may  also  distort  the  relation 
between  accounting  and  opportunity  costs.  Accounting  costs  for 
capital  goods  whose  expected  life  extends  beyond  a  single  accounting 
period  are  a  function  of  the  historical  cost  of  these  goods  and 
their  expected  life.  Changing  market  conditions  over  the  life 
of  a  capital  good  may  change  the  value  of  this  resource  in  its 
alternative  uses  and  therefore  its  opportunity  cost.  The  accounting 
cost  would  remain  unchanged,  however,  fixed  by  the  historical 
cost  of  the  capital  good  and  its  expected  life. 

As  an  example,  a  classroom  building  built  in  one  period 
would  be  valued  by  the  accountant  in  terms  of  its  historical 
cost  and  expected  life.  A  depreciation  schedule  would  be 
established  based  on  this  information.  If  conditions  later 
changed  such  that  no  alternative  uses  for  the  building  existed, 
the  building's  opportunity  cost  would  become  zero  while  its 
accounting  cost  would  remain  unchanged.  Alternatively,  if  the 
market  value  of  the  building  increased,  its  opportunity  cost 
would  also  increase,  but  the  accounting  cost  would  remain 


unchanged . 


Sunk  Costs 


Costs  that  have  already  been  incurred  and  cannot  be  recouped 
are  referred  to  as  sunk  costs.  Such  costs  should  not  be  included 
in  CTEA  cost  models.  Training  technologies  that  employ  off-the- 
shelf  technology  provide  an  example  of  sunk  cost.  The  Research 
&  Development  embedded  in  the  off-the-shelf  technology  represents 
a  cost  that  has  been  incurred,  and  in  most  instances,  this  cost 
cannot  be  recouped.  As  such,  the  R&D  should  be  treated  as  a 
sunk  cost  and  ignored.  The  key  in  this  example  is  that  the  cost 
cannot  be  recouped.  By  contrast,  however,  the  R&D  reguired  to 
develop  a  new  device  or  other  technology  that  cannot  be  bought  off- 
the-shelf  should  be  included  in  the  system  cost.  The  investment 
in  this  R&D  could  be  diverted  to  alternative  uses  and  is  therefore 
not  a  sunk  cost. 

Fixed  and  Variable  Costs 

CTEA  cost  models  distinguish  between  fixed  and  variable 
costs.  Fixed  costs  do  not  vary  with  the  number  of  trainees  or 
amount  of  training  provided  in  a  planning  period.  These  costs 
are  incurred  regardless  of  how  much  training  takes  place  during 
the  period.  This  follows  from  the  fact  that  the  resources  involved 
cannot  be  easily  diverted  to  other  uses  during  the  planning 
period.  They  are,  in  fact,  fixed  resources.  Depending  on  the 
length  of  the  planning  period,  some  resources,  such  as  classroom 
space,  may  be  fixed  in  supply.  Constructing  new  classrooms  or 


converting  old  ones  to  alternative  uses  cannot  be  done  on  short 
notice . 


Other  resources,  however,  can  be  varied  during  the  planning 
period  to  affect  the  number  of  trainees  or  amount  of  training 
provided.  The  cost  of  these  resources  is  considered  a  variable 
cost.  Examples  might  include  the  number  of  instructors  assigned 
to  a  school  or  the  instructional  materials  provided.  Fixed 
resources  define  the  scale  of  training  possible  in  any  planning 
period.  The  number  of  special  training  devices  provided,  for 
example,  or  the  number  of  classrooms  constructed  and  the  intensity 
of  their  use  defines  the  upper  limit  of  training  possible  during 
a  planning  period.  Variable  resources  can  be  applied  to  these 
fixed  resources  to  determine  the  amount  of  training  done  within 
this  limit. 

The  scale  of  training  possible  can  be  expanded  or  contracted 
in  subsequent  planning  periods  by  changing  the  stock  of  fixed 
resources  used.  In  the  long  run,  all  resources  become  variable 
resources.  Given  long  enough,  it  is  possible  to  construct  new 
classrooms  or  to  convert  existing  classrooms  to  alternative 
uses.  With  enough  time,  additional  training  devices  can  be 


provided.  In  the  short  run,  represented  by  a  planning  period, 
however,  some  resources  will  be  treated  as  fixed  and  others  as 
variable . 


Time  Value  of  Money 

The  life-cycle  cost  of  alternative  training  technologies 
will  normally  involve  incurring  different  costs  at  different 
times.  In  order  for  two  or  more  alternatives  to  be  compared  on 
an  equal  economic  basis,  CTEA  cost  models  must  consider  the 
costs  of  each  alternative  currently  or  at  their  "present  values." 
This  recognizes  that  money  has  earning  power  over  time.  To  find 
the  present  value  of  expected  future  costs,  the  technique  of 
discounting  is  used.  This  technique  determines  the  amount  of 
money  which,  if  invested  today  at  a  selected  interest  rate, 
would  be  sufficient  to  meet  expected  future  costs. 

Present  value  analysis  normally  will  not  be  needed  in 
evaluating  alternatives  that  do  not  involve  large  capital 
expenditures  and  whose  cash  flows  under  each  alternative  occur 
at  approximately  the  same  times  during  the  planning  period. 

Where  the  timing  of  expenditures  is  significantly  different, 
however,  present  value  analysis  should  be  used.  When  long-lived 
capital  assets  are  to  be  government-owned  or  built  to  government 
specification,  present  value  analysis  is  necessary  to  determine 
whether  or  not  it  is  economical  to  incorporate  features  in  the 
system  that  will  cost  more  initially,  but  which  will  reduce 
future  costs  for  operation,  maintenance,  repair,  and  improvements 

Discount  Rates 


Although  present  value  analysis  is  a  generally  accepted 
practice,  selecting  an  appropriate  interest  rate  for  discounting 


in  the  analysis  has  been  the  subject  of  much  controversy.  Arguments 
have  been  presented  for  using  the  cost  of  borrowing  by  the  Treasury 
to  using  the  rate  of  return  that  can  be  earned  in  the  private 
sector.  The  rate  applied  has  a  direct  effect  on  the  results  of 
an  analysis;  therefore,  the  choice  of  a  rate  is  very  important. 

The  present  value  of  two  training  technologies,  one  with  the 
cost  up  front  and  another  with  the  cost  deferred  into  the  future, 
will  differ  sharply  depending  on  the  discount  rate  used. 

A  high  discount  rate  reduces  the  cost  in  present  value 
terms  of  a  technology  whose  cost  is  deferred  into  the  future. 

The  higher  the  discount  rate  used,  the  more  advantage  given  to 
deferred  cost  technologies  in  comparison  with  up-front  cost 
technologies.  OMB  Circular  A-94  requires  executive  agencies, 
including  the  Department  of  Defense,  to  use  a  10  percent  discount 
rate  together  with  constant  dollars  for  determining  the  present 
value  of  expenditures  spread  over  time.  The  only  exception  is 
for  water  resource  projects  where  a  7  percent  discount  rate  is 
used . 


Constant  versus  Current  Dollars 


Changes  m  the  general  price  level  will  affect  the  purchasing 
power  of  the  dollar.  When  the  general  price  level  is  rising, 
the  real  value,  or  purchasing  power  of  the  dollar  is  falling. 
Conversely,  when  the  level  is  falling,  the  real  value  of  the 


dollar  is  rising.  There  are  two  basic  approaches  to  dealing 


with  price-level  changes  in  cost  studies.  (1)  Estimate  future 
costs  in  today's  prices;  that  is,  in  "constant  dollars."  (2) 
Estimate  future  costs  at  the  anticipated  price  levels  that  will 
exist  then;  that  is,  in  "current  dollars." 

Constant  dollars  are  always  associated  with  the  purchasing 
power  of  the  dollar  in  a  base  year.  Estimates  are  in  constant 
dollars  when  future  costs  are  adjusted  to  exclude  inflation  so 
that  they  reflect  the  level  of  purchasing  power  in  the  base 
year.  OMB  Circular  A-94  requires  executive  branch  agencies  to 
use  constant  dollars  in  their  cost  analyses. 


Residual  Value  of  Assets 

The  length  of  the  planning  period  may  exceed  the  expected 
life  of  some  resources  used  in  training  requiring  that  these 
resources  be  replaced  once,  even  several  times  during  the  training 
period.  In  other  cases,  however,  the  expected  life  of  a  resource 
may  exceed  the  length  of  the  planning  period.  At  the  end  of  the 
planning  period  the  resource  will  have  a  residual  value  which 
should  be  subtracted  from  its  cost.  An  example  might  be,  once 
again,  a  classroom  building  or  possibly  a  training  device.  To 
the  extent  a  resource  is  not  fully  expended  during  the  planning 
period,  its  cost  should  be  adjusted  to  reflect  this.  CTEA  cost 


models  will  consider  the  residual  value  of  the  resources  used. 


Indirect  Benefits 


Some  training  technologies  may  have  secondary  effects  which 
occur  outside  the  organization  that  provides  the  training.  An 
example  of  an  indirect  benefit,  and  one  not  normally  considered 
in  evaluating  military  training,  is  the  worth  of  the  training  to 
the  individual  in  preparing  him  or  her  for  a  civilian  occupation. 
The  value  of  this  training  in  the  broader  social  setting  is 
normally  not  considered  in  the  CTEA  cost  model  since  the  benefit 
is  not  realized  directly  by  the  provider  of  the  training.  In 
some  instances,  however,  the  analyst  may  attempt  to  document 
indirect  benefits  and  evaluate  them  outside  the  cost  model. 


SUMMARY 

An  economic  analysis  is  a  critical  step  in  the  design  of 
training  systems.  Following  Braby  (1975),  a  rational  choice  of 
an  instructional  delivery  system  cannot  be  based  upon  training 
effectiveness  without  regard  to  cost  and  vice  versa.  In  order 
to  facilitate  the  economic  analysis  of  instructional  delivery 
systems,  a  cost  model  must  be  constructed.  This  section  has 
reviewed  the  applications  and  objectives  of  cost  models  for  CTEA 
and  the  economic  concepts  employed  therein.  As  such,  it  provides 
a  framework  for  reviewing  the  CTEA  cost  literature  in  the  section 


to  follow. 


III.  A  REVIEW  OF  CTEA  MODEL  COST  LITERATURE 


MODEL  COMPARISON 

Following  an  approach  used  by  Rosen  et  al  (1981),  17  CTEA  cost 
models  and  guidebooks  are  compared  in  Table  1.  The  comparison 
includes  an  examination  of  how  the  various  economic  issues  described 
above  are  addressed  by  CTEA  cost  models.  Most  of  these  models  were 
developed  for  the  Army.  The  comparison  shows  substantial  variance 
in  the  treatment  of  inflation,  discounting,  and  residual  assets. 

Less  than  half  could  be  used  for  cost  prediction  in  the  conceptual 
stage  of  weapon  system  deveopment.  In  the  section  which  follows, 
each  model  and  guidebook  is  examined  in  greater  depth  with  respect 
to  these  and  other  features. 

REVIEW  AND  ASSESSMENT 

This  section  reviews  each  CTEA  cost  model.  The  review  traces 
the  conceptual  and  methodological  developments  in  CTEA  cost  modeling 
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which  have  taken  place  since  1975.  The  features  and  issues  examined 
include  the  following: 


--the  branch  of  service  for  which  model  was  developed, 

--the  objectives  of  the  model, 

--the  strengths  and  weaknesses  of  the  model, 

--the  link  established  between  cost  and  training  effectiveness, 
--the  field  testing  of  the  cost  model,  if  done,  and 
--the  suitability  of  the  model  for  CTEA  cost  prediction  in 
the  early  development  stage  of  a  weapon  system 

Training  Effectiveness  Cost  Effectiveness  Prediction,  (TECEP) 

This  technique  for  choosing  cost-effective  instructional 
delivery  systems  was  developed  for  the  Navy  by  Braby  et  al  of  the 
Training  Analysis  and  Evaluation  Group ( TAEG )  at  the  Naval  Training 
Equipment  Center  in  Orlando,  Florida.  This  1975  model  was  the 
capstone  of  several  earlier  TAEG  efforts  and  contains  refinements 
incorporated  during  a  year  of  field  trials.  The  TAEG  group  worked 
with,  borrowed  from,  and  synthesized  much  prior  work  of  media 
selection  experts, task  category  and  learning  guidance  developers, 
and  inter-service  training  designers.  Matlick  et  al  (1980a)  describe 
the  TECEP  cost  model  as  the  most  thorough  application  in  the  Life 
Cycle  System  Management  System. 

The  resulting  model  is  the  foundation  for  much  of  the  military 
cost  and  training  effectiveness  efforts  of  the  last  decade.  As 
noted  in  Rosen  et  al  (1981)  and  Knerr  et  al(  1984a),  the  Army  and  the 
Navy  have  incorporated  this  model  into  their  Instructional  System 


Development  (ISD)  strategies.  Knerr  et  al  further  note  that  it 
has  been  applied  extensively  since  being  incorporated  in  the 
military  ISD  model. 

The  model  was  developed  to  assist  the  skilled  training  system 
designer  to  make  instructional  delivery  system  choices  during  the 
conceptual  design  phase  of  a  weapon  system.  The  system  designer 
identifies  training  objectives,  and  chooses  two  or  more  alternative 
instructional  delivery  systems  capable  of  supporting  the  objectives. 
Estimating  the  costs  of  these  alternative  delivery  systems  is  the 
next  step  in  the  TECEP  process.  The  model  is  designed  to  develop 
training  programs  for  institutional  training. 

The  37  cost  elements  in  the  TECEP  model  incorporate  the  cost  of 
acquiring  and  operating  facilities  and  equipment,  the  cost  of 
supplies,  the  cost  of  the  design  of  instructional  material,  and  the 
cost  of  student  and  support  personnel.  Cost  estimates  are  generated 
for  each  alternative  and  comparisons  are  made  of  the  advantages  and 
disadvantages  of  each  system. 

The  cost  elements  are  well-defined,  the  algorithm  is  easily 
understood, and  at  the  time  of  its  development  the  costing  portion  of 
the  TECEP  model  was  computerized  using  a  FORTRAN  IV  program.  This 
model  was  originally  developed  for  the  Navy  and  included  naval  data 
sources  and  references.  It  has  since  been  modified  for  use  with 
Army  sources  and  references.  The  cost  output  of  the  TECEP  procedure 
included  the  present  value  for  each  alternative,  the  total  and 
average  annual  cost  per  student  position,  and  the  distribution  of 
the  incidence  of  costs  over  the  life  of  each  alternative.  Other 
variables  may  now  be  in  the  output. 


The  TECEP  model's  emphasis  is  on  cost  minimization.  It  can  be 
used  to  compare  alternative  training  approaches.  It  is  not  intended 
to  assess  differential  benefits  or  effectiveness  for  these 
approaches.  The  training  approaches  for  which  costs  are  estimated 
are  not  presented  in  any  ranked  order.  The  model  cannot  forecast 
total  system  cost. 

The  TECEP  model  requires  access  to  multiple  data  sources,  and 
there  will  be  a  variability  in  data  reliability.  This  fact  should 
be  noted.  It  assumes  all  variable  cost  functions  are  linear  --  an 
assumption  that  may  not  be  tenable  for  specific  training  situations. 
The  model  does  not  provide  any  means  for  evaluating  secondary  or 
spillover  effects.  It  assumes  these  effects  to  be  constant  for  all 
alternatives . 

The  authors  of  TECEP  provide  some  common  sense  considerations: 
(a)  if  an  alternative  involves  a  low  cost  per  student  graduate  but  a 
significant  initial  investment  --  such  as  computer-assisted 
instruction  --  it  makes  little  sense  to  consider  this  technique 
unless  resources  appear  in  existing  budgets  (b)  similarly,  if 
courseware  is  to  be  locally  developed,  then  skilled  personnel, 
equipment,  time  and  dollars  must  be  available  (c)if  command  policy 
or  existing  investment  in  production  facilities  would  require 
significant  change  for  the  proposed  alternatives,  then  these 
practical  factors  will  outweigh  any  model-recommended  alternatives. 

As  published,  the  cost  elements  for  instructional  material 
development  would  require  modifications  to  accurately  reflect  the 
cost  of  computer-assisted  instruction.  If  the  analyst  is  required 


to  provide  cost  by  task,  the  model  would  need  modification.  The  cost 
model  is  based  on  valid  economic  concepts. 

As  noted  earlier,  the  TECEP  model  has  been  the  basis  for  much  of 
the  training  and  cost  evaluation  in  the  services  since  1975.  Its 
authors  recognize  the  limitations  of  the  model  in  both  its  training 
and  costing  portions  and  note  that  testing  is  continuing.  Its  most 
significant  contribution  is  as  the  seedbed  for  the  training  and  cost 
models  which  have  since  evolved. 

Training  Efficiency  Estimation  Model ,( TEEM) : 

This  model  was  developed  at  the  Army  Research  Institute  by 
Jorgensen  and  Hoffer  in  1978  as  part  of  the  continuing  effort  to 
refine  training  estimation  during  the  conceptual  stage  of  the  weapon 
system  development. 

TEEM's  unique  contribution  to  military  model  evolution  is  in  the 
development  of  a  training  efficiency  ratio  and  the  linking  of  that 
ratio  to  cost  to  provide  cost  effectiveness  ratios  as  the  basis  for 
choice . 

The  methodology  can  be  used  at  the  earliest  stages  of  weapon 
system  development  where  the  analyst  is  dealing  with  probable  weapon 
characteristics  from  which  are  developed  preliminary  duty  or  task 
description  lists.  Matlick  et  al  do  not  feel  the  ratio  is  strong  but 
probably  the  "best  available  in  an  early  CTEA"  ( 1980 , a , p . V- 12 ) .  TEEM 
was  developed  for  use  in  an  institutional  setting. 

The  model  generates  a  variety  of  training  programs  combining 


different  media  and  methods  to  meet  the  training  needs  of  the 
preliminary  task  description.  Generated  along  with  other  training 


programs  is  an  ideal  training  program  with  no  constraints  to  be  used 
as  the  base  case.  The  ideal  training  program  and  constrained 
training  programs  are  quantified.  An  efficiency  ratio  is  derived  by 
dividing  the  constrained  programs  by  the  ideal.  Efficiency  here 
refers  to  the  extent  that  task  characteristics  match  training 
program  character istics .  The  efficiency  ratio  reflects  the  match 
between  learning  needs  and  the  media  and  methods  meeting  those 
needs . 

A  decision  metric  is  derived  by  dividing  the  cost  of  each 
alternative  by  its  efficiency  ratio.  The  costing  is  based  on  the 
TECEP  cost  model  then  in  use  for  several  Army  training  cost  estimate 
efforts.  The  training  efficiency  ratio  has  since  been  modified  to 
use  actual  equipment  as  the  base  case  and  to  use  an  Army-modified 
TECEP  cost  model. 

A  significant  advantage  of  the  TEEM  ratios  is  that  they  can  be 
used  to  compare  alternative  training  programs  for  a  given  weapon 
system  as  well  as  providing  comparisons  across  weapon  systems.  Rosen 
et  al  (1981)  consider  the  TEEM  model  highly  useful  since  it  is 
predictive  of  effectiveness,  efficiency,  and  cost-ef f ectiveness . 

A  drawback  of  the  original  model  is  the  exclusion  from 
consideration  of  such  information  as  managerial  constraints, 
personnel  characteristics  and  teacher  characteristics .  The  model 
also  needed  further  testing  to  resolve  the  question  of  what  weights 
should  be  applied  to  the  cost  factors. 
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Training  Consonance  Analysis ,  ( TCA) : 

This  model  was  designed  for  the  Army  Research  Institute  by 
Hawley  and  Thomason  in  1978  to  perform  CTEA  at  the  points  required 
by  the  Life  Cycle  System  Management  Model.  The  authors  further 
modified  the  TECEP  and  TEEM  descriptions  of  tasks  and  media-method 
combinations  to  arrive  at  Training  Consonance  Ratios.  The  TCR  is 
the  sum  of  matches  between  the  presence  of  a  variable  in  a  task 
description  and  in  a  medium-method  combination  divided  by  the  number 
of  variables  in  the  task  description. 

For  example,  if  43  training  consonances  are  scored  and  the 
task  description  includes  48  variables,  then  the  ratio  for  that 
task  is  .896. 

The  methodology  also  uses  training  deficiency  and  training 
excess  scores  to  remove  inefficient  training  methods.  The  TCR 
can  be  used  with  the  TECEP  costing  variables  to  provide  cost- 
effective  ratios.  It  is  designed  for  institutional  training 
projections . 

In  addition  to  providing  Army-related  performance  data  source 
suggestions,  Hawley  and  Thomason  provide  important  advice  to  the 
cost  analyst  to  include  in  their  cost  data  requests,  at  a  minimum, 
(a)  a  description  of  training  concepts  which  describes  how  training 
will  be  carried  out  and  what  instructional  methods  are  to  be 
employed  (b)  the  make,  model,  quantity,  and  quality  of  materiel 
required  (c)  the  quantity,  grade,  and  skill  level  of  personnel 
required  and  (d)  time  requirements  and  materiel  usage  data  such  as 
the  length  of  course  and  supplies  expended. 
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Another  contribution  to  the  costing  methods  developed  earlier 
were  their  recommendations  to  project  sunk  costs,  fixed  costs, 
and  variable  costs  for  use  in  alternative  comparisons.  Hawley 
and  Thomason  advise  that  training  resource  costs  be  exp-  ;ssed  in 
constant-year  dollars  and  that  DOD  policy  specifies  that  for 
that  for  CTEA  the  base  cost  year  is  the  fiscal  year  following  the 
calendar  year  in  which  the  study  is  scheduled  to  be  completed. 
Matlick  et  al  (1980a)  interpret  this  to  mean  that  the  base  cost  year 
is  the  year  the  full-scale  or  limited  production  of  the  materiel 
system  is  projected  to  begin  and  are  troubled  by  the  lack  of 
citation  for  the  policy. 

An  important  consideration  for  any  cost  analyst  in  making 
projections  is  the  possibility  that  the  assumptions  on  which  the 
model  is  based  may  change.  Sensitivity  analysis  allows  projections 
to  be  made  of  cost  and  training  effectiveness  changes  if  the  values 
of  certain  assumptions  change.  Here  again  Hawley  and  Thomason  offer 
important  suggestions  to  the  analyst  recommending  that  training 
equipment  reliability,  availability  and  maintainability  assumptions 
be  allowed  to  vary  for  the  cost  projections.  They  also  recommend 
considering  such  changes  as  a  switch  from  military  to  civil  service 
instructors  or  from  enlisted  personnel  to  officers  in  predicting 
instructor  salaries.  Performing  these  analyses  would  allow  the 
analyst  to  select  training  methodologies  that  are  cost-effective  for 
most  of  the  range  of  likely  values  for  these  assumed  parameters. 

This  model  expands  the  TECEP  costing  considerations  by  alerting 
the  cost  analyst  to  provide  the  fullest  training  description  in  cost 
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data  requests  and,  through  the  use  of  sensitivity  analysis,  to 
examine  changes  in  the  values  of  assumptions.  Knerr  et  al  (1984a) 
further  note  that  it  can  be  used  to  evaluate  individual  tasks, 
groups  of  tasks  or  the  entire  program. 

Litton  Mellonics  Training  Cost  Model: 

Matlick  et  al  of  Litton  Mellonics  reviewed  the  relationships 
between  the  development  of  Army  weapon  systems  and  the  concomitant 
training  requirements  for  the  ARI  Field  Unit  at  Fort  Bliss,  Texas 
and  published  their  work  in  1980.  Their  objectives  were  to  provide 
Army  analysts  with  a  performance  guide  for  CTEA  at  each  stage  of  the 
Life  Cycle.  (Matlick  et  al,  1980b) 

They  reported  that  despite  Army  doctrine,  many  systems  were 
fielded  with  most,  some,  little,  or  none  of  the  data  required  for 
training  decisions.  Their  efforts  resulted  in  the  development  of  an 
analogous  task  method  which  proceeds  task-by-task  to  extrapolate 
effectiveness  and  cost  of  current  training  of  tasks  similar  to  those 
required  for  the  new  system. 

They  also  developed  a  process  to  evaluate  task  criticality  to 
highlight  trainability  concerns  for  further  research.  And,  finally, 
they  made  significant  modifications  in  the  TECEP  cost  model  to  (a) 
make  it  truly  Army-compatible  and  (b)  to  include  the  cost  of  unit 
training  if  proficiency  had  not  been  attained  at  the  institutional 
level . 

This  model  further  developed  the  TECEP  variables  by  modifying 
the  institutional  training  elements  to  incorporate  both  their 
classroom  and  field  cost  elements. 


The  most  significant  part  of  the  Litton  modifications  to  the 


TECEP  equations  is  the  provision  of  specific  sources  of  information 
for  the  cost  data.  The  authors  have  provided  the  Army  equivalent  of 
the  original  Navy  information  sources  in  a  step-by-step  Performance 
Guide . 

Like  TECEP,  the  Litton  model  is  intended  to  aid  the  C TEA  analyst 
to  prepare  recommendations  regarding  relative  choices  among 
alternatives.  It  cannot  be  used  for  budgetary  purposes. 

Nevertheless,  it  is  important  in  collecting  data  on  training  costs 
for  the  analyst  to  be  familiar  with  the  Five  Year  Defense  Plan  and 
its  associated  budget  programs. 

An  advantage  of  the  Litton  model  is  the  provision  of  discounted 
cost  figures  either  by  task(s)  or  by  training  program.  The  authors 
of  the  performance  guide  also  provide  useful  instructions  on  the 
inclusion  of  both  direct  and  indirect  costs  of  institutional  training 

The  costing  algorithm  presents  some  problems.  Like  TECEP,  the 
instructional  material  development  category  requires  refinement  to 
cost  computer-assisted  instruction.  The  model  also  assumes  the 
instructor  to  student  ratio  to  be  constant  from  year  to  year.  This 
ratio  is  then  used  to  estimate  facilities  needed  and  to  cost  these 
facilities  over  the  planning  period.  Self-paced  instruction  would 
require  a  different  approach. 

As  noted  earlier,  the  major  contribution  of  the  Litton  model  is 
the  provision  of  very  specific  Army  data  sources.  The  performance 
guide  is  designed  for  use  by  Army  personnel  who  are  familiar  with, 
but  not  necessarily  experts  in,  Army  training  and  task  analysis.  It 
also  does  not  require  the  use  of  psychologists  or  mathematicians.  By 


providing  the  non-expert  analyst  with  a  variety  of  approaches  to 


data  situations,  it  increases  the  number  of  personnel  available  to 
perform  CTEA.  Knerr  et  al  (1984a)  recommend  the  Litton  model  for  all 
CTEA  situations  in  the  Life  Cycle  System  Management  Model. 

TRADOC  Systems  Analysis  TEA  Handbook  (TRASANA): 

This  1980  draft  handbook  was  developed  by  TRADOC ' s  Systems 
Analysis  Agency  at  the  White  Sands  Missile  Range.  Rosen  et  al  (1981) 
state  that  the  handbook  is  envisioned  as  an  evolutionary  document 
with  the  ultimate  goal  of  providing  a  "how  to"  guidebook.  As 
outlined  in  this  draft,  the  methodology  describes  what  should  be 
done  to  carry  out  an  analysis  but  is  not  specific  on  how  to  do  it. 

The  handbook  advises  that  an  analyst  must  insure  (a)  that  all 
costs  for  support  of  the  training  subsystem  are  included  (b)  that 
total  costs  are  captured  to  allow  for  valid  comparisons  and  (c)  that 
there  be  close  coordination  between  the  cost  analyst  and  the 
effectiveness  analyst.  The  handbook  contains  useful  prompts  and 
reminders  for  the  analyst  as  the  CTEA  proceeds. 

The  handbook  recognizes  that  raw  cost  data  received  from  other 
agencies  will  require  analysis  and  transformation  into  more  useful 
form.  The  handbook  was  designed  to  cover  all  phases  of  the  system 
life-cycle,  not  just  the  development  stage.  Its  stated  intention 
would  provide  cost  estimates  for  both  institutional  and  unit 
training. 

In  contrast  to  the  models  discussed  earlier,  the  handbook 
recommends  that  a  variable  cost,  variable  effectiveness  model  is 
the  most  appropriate  for  comparing  alternative  training  systems. 

The  handbook  states  that  neither  variable  has  complete  meaning 


until  both  are  considered  together. 


No  cost  model  is  provided  by  the  handbook  other  than  the  very 
general  equation  that  Total  Training  Costs  equals  Institutional 
Training  Costs  plus  Unit  Training  Costs.  In  fact,  the  authors 
recommend  that  cost  methodology  be  "generalized"  to  allow  changes  as 
the  analysis  progresses. 

The  handbook  provides  useful  sources  of  cost  data  information  in 
addition  to  those  provided  in  the  Litton  Mellonics  performance 
guide.  It  also  includes  a  cost  term  definition  appendix  which 
provides  helpful  advice  on  procurement  schedule  flexibility 
recommending  that  different  procurement  quantity  schedules  be 
assessed  for  both  their  cost  results  and  their  training 
effectiveness.  It  also  advises  that  equipment  cost  data  requests 
include  a  probable  "size  of  buy"  with  minimum  and  maximum  parameters 
so  that  the  cost  estimator  can  establish  the  appropriate  unit  cost. 

A  training  cost  data  appendix  provides  useful  definitions  and 
examples  for  elements  of  institutional  training,  unit  training, 
training  system  requirements  and  life-cycle  costs.  As  noted  by 
Matlick  et  al  (1980a),  there  is  no  explicit  provision  for  either 
discounting  or  the  evaluation  of  residual  assets. 

The  handbook ' s  section  on  analysis  presentation  provides  some 
important  advice.  Noting  that  decision  makers  are  as  interested  in 
the  major  cost  elements,  expenditure  schedules  and  other  aspects  as 
they  are  in  total  cost  of  a  training  system,  they  advise  that  care 
be  taken  to  insure  that  vital  ingredients  of  the  analysis  are  not 
obscured  in  some  final  cost  figure.  The  recommended  output  would 
highlight  sub-elements  of  high  impact  on  total  costs  as  well  as 


those  whose  incremental  or  decremental  change  would  influence  system 
effectiveness.  The  total  numbers  would  be  appended  to  this 
highlighted  analysis. 

As  presented  in  this  1980  draft,  the  handbook  could  be  used  with 
a  specified  cost  algorithm  to  arrive  at  a  more  complete  cost 
prediction. 

General  Model/Methodology  for  Conducting  Cost  and  Training 
Effectiveness  Analyses: 

An  earlier  effort  to  develop  a  TRADOC  handbook  was  produced  in 
1976  by  Vector  Research,  working  as  a  subcontractor  to  BDM  Services. 
As  noted  by  Matlick  et  al  (1980a),  the  guide  outlines  a  process 
rather  than  providing  a  procedure  or  method  whereby  inputs  lead  to 
precisely  defined  outputs. 

The  approach  developed  by  Vector  is  unique  and  does  not  appear 
to  be  a  derivative  of  any  of  the  other  approaches  examined.  They 
examined  training  for  a  number  of  Army  systems  and  identified  eight 
unique  systems:  large  group  war  games  simulators,  individualized 
and/or  small  group  lesson  delivery  systems,  small  group  practical 
maneuver  and  deployment  instructional  games,  program  directed  hands- 
on  job  performance  aids,  large  weapon  system  practice  firing 
adapters,  small  group  combat  engagement  simulators,  trouble  shooting 
training  simulators,  and  small  weapon  system  practice  firing 
adapters.  They  then  developed  weighted  measures  of  training 
effectiveness  for  a  base  system  and  each  alternative. 


The  cost  analysis  is  extremely  simple.  The  analyst  selects  a 
sample  system  most  similar  to  the  alternatives  to  be  evaluated  and 


calculates  a  per  unit  cost  based  on  the  expected  life-cycle  number 
of  units  used.  The  costs  of  the  alternatives  are  multiplied  by  the 
fraction  resulting  from  dividing  the  effectiveness  of  the  base  case 
by  the  effectiveness  of  the  alternative.  This  figure  is  then 
compared  to  the  cost  of  the  base  case.  All  costing  follows  training 
effectiveness  determination. 

Matlick  et  al  (1980a)  state  that  the  alternatives  could  also  be 
compared  with  each  other  in  which  case  the  second  alternative  would 
replace  the  base  case  in  the  numerator. 

As  noted  by  Matlick  et  al  (1980a),  this  methodology  provides  no 
guidance  on  front  end  analysis.  The  decision  step  linking  a  training 
need  to  one  of  the  eight  training  methods  is  missing.  They  also  note 
that  it  can  only  be  used  for  actual  training  programs  where  interval 
or  ratio  data  are  available. 

It  has  nothing  to  offer  in  terms  of  training  cost  prediction  in 
early  weapon  development.  Its  costing  formula  does  not  reflect 
either  discounting  or  the  consideration  of  residual  assets.  It  is 
unclear  how  student  and  instructor  costs  are  calculated.  Further 
reviews  do  not  indicate  whether  the  handbook  was  ever  used  by 
TRADOC . 

Training  Developers  Decision  Aid  for  Machine -Ascendant  Military 


OccuDational  SDecialties  (TDDA/MAMOS) 


Specialties  are  to  be  trained.  TDDA/MAMOS  assumes  a  usable  task 
list . 

The  four  functional  elements  of  TDDA/MAMOS  are:  task 
description,  training  prescription,  training  hierarchies  and 
seguence,  and  training  costs.  Matlick  et  al  (1980a)  note  that 
the  training  prescriptions  are  modifications  of  the  TECEP  and 
TEEM  approaches.  The  costing  is  unique  --  and  complicated. 

This  is  not  a  weapon  system-based  model.  It  is  based  on  the 
Military  Occupational  Specialty,  with  the  tasks  for  each  MOS  treated 
as  a  single  set  --  some  taught  in  schools,  some  on-the-job,  and  a 
few  given  no  formal  training.  After  tasks  are  assigned  to  resident 
training  or  on-the-job  training,  the  model  sets  up  a  relative  cost 
formula  for  feasible  alternatives  using  a  cost-rating  technique. 

According  to  Matlick  et  al  (1980a),  this  cost  rating  technique 
was  justified  by  the  authors  because  (a)  actual  dollar  costs  vary 
over  time  while  relative  differences  remain  essentially  constant  (b) 
ratings  can  include  items  such  as  student  time  without  conversion  to 
dollar  amounts,  and  (c)  the  various  methods  of  instruction  can  be 
ranked  on  relevant  cost  dimensions.  Rosen  et  al  (1981)  also  note 
that  the  model  was  attempting  to  overcome  inflation  uncertainties 
and  the  consequent  time-consuming  data  conversions.  It  is  not  clear 
how  the  following  formulation  would  save  time. 

For  classroom  instruction,  TDDA/MAMOS  divides  the  following  ten 
training  methods  into  seven  method  cost  classes:  conventional, 
demonstration,  case  study,  guided  discussion,  peer  tutor,  tutor, 
programmed  instruction,  traditional  practical  exercise,  programmed 
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practical  exercise  and  computer-assisted  instruction.  Then  the 
direct  and  indirect  cost  variables  were  categorized  into  seven 
classes:  square  footage,  instructor  to  student  ratio,  system 
equipment,  furnishing,  expendable  supplies,  training  aid 
development,  and  training  material  development.  Finally,  the  method 
cost  classes  were  ranked  on  each  of  the  cost  variables,  resulting  in 
a  seven-by-seven  matrix  of  values. 

A  mean  rank  is  then  calculated  for  each  method  cost  class  and  a 
mean  rank  cost  multiplier  is  also  developed.  Then  the  number  of 
hours  of  each  method  within  the  training  cost  option  is  multiplied 
by  its  cost  factor  to  get  the  Method  Cost  Indicator.  This  MCI  for 
all  methods  within  the  training  option  is  then  summed  yielding  a 
Resident  Option  Cost  Indicator. 

with  the  exception  of  computer-assisted  instruction,  the  cost 
method  classes  and  cost  variables  were  also  calculated  for  On-The- 
Job  training  to  obtain  an  OJT  Option  Cost  Indicator.  The  two  were 
then  summed  to  produce  a  Training  Option  Cost  Indicator. 

This  TOCI  is  then  used  to  establish  a  ratio  of  one  training 
option  to  another,  giving  the  training  manager  an  indication  of 
relative  cost. 

Matlick  et  al  note  in  their  1980  review  that  TDDA/MAMOS  had  not 
yet  been  fully  developed  and  would  require  an  extensive  validation 
effort  to  demonstrate  its  usefulness  and  the  precision  of  its 
outputs.  In  their  1981  review,  Rosen  et  al  indicate  that  TDDA/MAMOS 
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As  noted  earlier,  this  model  would  not  be  useful  in  the 
development  stage  of  a  weapon  system  and  has  no  relation  to  the 
life-cycle  costing  of  a  system.  Even  for  its  stated  purpose  of 


devising  different  training  strategies  for  Machine-Ascendant 
Military  Occupational  Specialities,  its  formula  for  developing  a 
relative  cost  indicator  is  intimidating. 

Multiple  Launch  Rocket  System  Repairer  Cost  and  Training 
Effectiveness  Analysis  (TRASANA  TEA  MLRSR) : 


This  CTEA  was  conducted  by  the  Army  TRAD OC  Systems  Analysis 
Activity  (TRASANA)  in  partnership  with  the  Army  Missile  and 
Munitions  Center  and  School  (MMCS)  in  1984  to  evaluate  the  relative 
cost  and  training  effectiveness  of  four  training  alternatives 
generated  by  the  MMCS  to  train  135  students  per  year.  MMCS  is  the 
proponent  for  individual  training  required  to  support  the  Multiple 
Launch  Rocket  System. 

The  major  hardware  subsystem  component  of  the  MLRS  is  a  tracked, 
self-propelled  launcher  loader ( SPLL ) .  The  original  eleven  week 
Program  of  Instruction  ( POI )  anticipated  training  135  students  per 
year  with  ten  SPLLS  for  hands-on  experience.  Only  three  SPLLS  were 
permanently  assigned  to  the  MMCS  with  a  fourth  on  a  more  or  less 
permanent  loan.  This  CTEA  was  conducted  because  of  the  equipment  gap 
and  anticipated  obtaining  training  device! s)  to  fill  that  gap.  It  is 
included  here  as  an  example  of  the  effort  required  when  the  in-place 
training  devices  are  other  than  those  projected  in  the  original 
Program  of  Instruction  and  as  an  illustration  of  incorporating  TECEP 


and  TRASANA  TEA  handbook  instructions. 


MMCS  generated  several  combinations  of  SPLLs  and  anticipated 
training  devices  and  included  the  original  plan  as  the  baseline. 
Subject  matter  experts  were  used  to  rank  the  effectiveness  of  the 
baseline  and  the  alternatives.  The  five  methods  were  compared  on 
their  research  and  development,  investment,  operating  and 
maintenance,  and  instructor  costs.  The  costs  reflect  the  equipment 
use  and  instructor  hours  for  blocks  of  instruction  which  require 
equipment . 

The  cost  estimates  combined  both  actual  experience  and 
comparisons.  For  example,  the  MMCS  used  nine  months  of  experience 
with  four  SPLLs  to  estimate  operating  and  maintenance  costs.  The 
operating  and  maintenance  costs  for  the  trainer  were  estimated  by 
the  project  manager's  office  who  decided  that  student  and  instructor 
stations  were  equivalent  to  the  cost  of  the  fire  control  panel 
trainer  used  at  Fort  Sill. 

TRASANA  computed  instructor  costs  for  baseline  and  alternatives 
by  estimating  the  maximum  and  minimum  number  of  instructor  hours 
needed  for  each  block  of  instruction.  The  variability  of  instructor 
costs,  operating  and  maintenance  costs  and  the  variability  of 
electrical  power  costs  for  the  trainer  led  to  the  presentation  of 
the  data  in  a  high  total  cost  and  low  total  cost  format  for  the 
baseline  and  each  alternative.  This  follows  the  guidelines  in  the 
TRASANA  TEA  handbook  which  call  for  presenting  the  sub-elements 
which  have  the  highest  impact  on  costs.  Student  costs,  and  costs 
for  buildings  and  instructors  in  a  classroom  were  considered  equal 
for  the  baseline  and  alternatives  and  were  factored  out,  reflecting 
the  TECEP  approach. 


Another  table  presented  the  ranked  cost  comparisons  alongside 
the  ranked  training  effectiveness  comparisons.  Again,  this  is  the 


tradeoff  display  recommended  in  the  TRASANA  TEA  handbook. 

This  CTEA  was  conducted  to  assess  training  effectiveness  and 
costs  in  an  institutional  setting  after  a  training  equipment  gap 
developed.  The  comparison-based  costing  estimate  for  the  trainer 
which  had  yet  to  be  developed  would  provide  some  guidance  for  the 
development  stage  of  a  weapon  system.  It  would  not  provide  budgeting 
information. 

Comparison-Based  Prediction  of  Cost  and  Effectiveness  of 
Training  Devices:  A  Guidebook ,( CBP ) 

This  guidebook  was  developed  for  the  Army  Research  Institute  by 
Klein  Associates  in  1985  in  an  effort  to  assist  the  Army  in  the 
evaluation  of  training  devices  early  in  the  weapon  acquisition 
cycle.  According  to  its  author,  it  represents  a  refinement  and 
improvement  of  the  comparability  analysis  approach  which  has  been 
used  by  the  Air  Force  since  the  1970  's  to  predict  equipment 
reliability  for  purchasing,  manpower  projections  and  downtime 
forecasts . 

The  methodology  is  not  a  derivative  of  any  of  the  previously 
discussed  models.  The  author  argues  that  a  technique  is  needed  that 
is  between  unstructured  expert  judgments  and  complex  data-driven 
models.  The  author  criticizes  unstructured  expert  judgments  as 
subjective,  difficult  to  evaluate  or  to  justify  to  others.  On  the 
other  hand,  he  notes  that  complex  models  require  data  that  are  not 
available  to  feed  decisions  early  in  the  life-cycle. 


The  CBP  solution  to  these  problems  involves  structuring  expert 
opinion,  using  data  available  on  similar  cases,  and  providing  an 
audit  trail  of  these  judgments  so  that  they  can  be  evaluated, 
compared  with  other  predictions,  and  adjusted  as  the  design  process 
advances . 

Describing  CBP  as  a  method  of  reasoning  by  analogy,  Klein  uses 
the  illustration  of  a  potential  homeseller  turning  to  an  appraiser 
for  assistance,  having  the  appraiser  identify  both  the  important 
factors  that  influence  price  and  similar  recently-sold  homes.  Klein 
likens  the  appraiser's  documented  estimate  to  a  subject  matter 
expert  who  provides  the  training  developer  with  a  recommendation 
based  on  important  causal  factors  with  sufficient  documentation  to 
provide  an  audit  trail  for  evaluating  the  decision.  In  a  section 
entitled  "Setting  up  the  Problem",  Klein's  analyst  has  decided  on  a 
proposed  training  device  description,  measures  of  training 
effectiveness,  and  important  causal  factors.  The  analyst  then 
examines  existing  devices  to  see  if  there  is  a  match.  It  is  not 
clear  from  the  guidebook  why  a  particular  training  device  is  being 
considered  for  the  new  weapon  system,  how  the  training  effectiveness 
measures  were  developed,  or  how  the  causal  factors  were  selected. 

After  finding  a  similar  training  device  or  devices,  the  analyst 
then  arranges  to  interview  the  subject  matter  expert(s)  on  the 
device(s).  Klein  places  great  emphasis  at  this  point  on  drafting  a 
guide  for  the  interview  so  that  identical  descriptions  will  be  given 
to  each  subject  matter  expert  on  the  proposed  device.  The  subject 
matter  expert  is  asked  to  compare  the  system  with  which  he  is 
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familiar  with  the  analyst's  proposed  device  and  features  and  to  make 
comparative  effectiveness  or  comparative  cost  predictions.  A  written 
record  is  maintained  of  this  interview. 

Some  problems  may  occur  during  this  interview.  The  analyst  has 
already  decided  on  the  comparison  case  which  is  how  the  subject 
matter  expert  (SME)  is  selected.  That  step  also  included  establishing 
measures  of  training  effectiveness  and  causal  factor  decisions. 

Klein  notes  that  the  SME  may  suggest  a  better  comparison  device, 
other  causal  factors,  or  better  Measures  of  Training 
Effectiveness  (MOTE)  based  on  experience.  This  would  require  a 
new  comparable  device  SME  interview,  and  a  new  round  of  causal 
factors  and  MOTEs.  Klein  forecasts  this  iterative  process  with  a 
circular  flowchart,  but  does  not  indicate  at  what  point  the 
reconsideration  and  reinterviewing  would  cease. 

Klein  recommends  several  alternative  strategies  for  structuring 
the  comparison-based  application  depending  on  time  constraints, 
abundance  of  comparison  cases,  availability  of  data,  and 
identification  of  SMEs. 

These  involve:  (a)  a  "global"  strategy  based  on  interviewing  one 
SME,  asking  for  a  judgment  on  the  proposed  device  based  on  the  SME's 
experience  with  the  comparable  device,  (b)  a  "high  driver"  strategy 
involves  asking  the  SME  to  detail  how  the  target  case  and  the 
comparison  case  differ,  and  how  much  each  of  the  high  drivers 
affects  the  difference,  (c)  the  "multiple  comparison  case"  strategy 
which  increases  confidence  if  the  same  prediction  is  reached 
independently  through  the  use  of  different  comparison  cases.  He 


suggests  this  strategy  could  also  involve  the  judgments  of  multiple 
SMEs,  that  if  these  multiple  judgments  converge,  the  confidence 
level  is  raised.  He  does  not  discuss  what  to  do  if  there  is  multiple 
judgment  divergence,  (d)  He  then  suggests  a  "cumulative”  strategy, 
adding  SMEs  until  there  is  enough  agreement  to  "feel  confident".  If 
there  are  disagreements,  Klein  feels  the  audit  trail  will  reveal  the 
basis  for  the  dif ference .( e ) Klein  finally  states  that  if  there  is 
one  SME  whose  judgment  is  valued  most  highly,  then  that  is  the  SME 
to  be  used. 

It  is  not  clear  how  these  subjective  approaches  differ  from 
those  Klein  has  criticized  and  is  attempting  to  replace.  Klein’s 
strongest  recommendation  for  the  CBP  method  is  the  keeping  of  a 
written  record,  the  so-called  "audit  trail"  for  the  decision.  He 
states  that  if  the  prediction  is  found  to  be  inaccurate  once 
operational  data  are  obtained  for  the  target  case,  the  audit  trail 
provides  "an  opportunity  to  identify  which  considerations  (causal 
factors)  were  responsible  for  the  mis judgment . "  (1985,p.A-4)  It  is 
not  clear  what  steps  follow  identifying  these  mis j udgments . 

The  audit  trail  suggested  here  and  by  others  can  be  a  valuable 
tool,  but  it  would  not  address,  in  this  instance,  selecting  the 
wrong  target  training  device  in  the  first  place. 

There  is  a  suggestion  throughout  the  Klein  methodology  that  a 
new  weapon  system  will  be  a  replacement  for  a  current  system,  and 
that  training  devices  for  the  new  system  will  involve  only 
incremental  changes  from  existing  devices.  The  approach  does  not 
involve  assessing  present  methods  and  devices  for  inadequate 


training.  In  reviewing  a  1982  version  of  this  methodology,  Knerr  et 
al  (1984a)  note  that  it  has  promise  if  the  information  of  interest  is 
the  relative  merits  of  potential  devices. 


There  is  no  clear  link  between  training  effectiveness  and  cost 
in  this  method.  The  SME  is  asked  to  make  a  relative  judgment  about 
a  pair-wise  combination  for  either  effectiveness  or  cost  changes. 

The  approach  could  be  used  to  refine  those  steps  in  more  complete 
methodologies  where  estimates  must  be  used. 

Cost-Effectiveness  Specification  for  Computer-Based 
Training  Systems: 

This  methodology  was  developed  by  Seidel  and  Wagner  of  the  Human 
Resources  Research  Organization  for  the  Defense  Advanced  Research 
Projects  Agency  and  published  in  1977.  It  was  designed  to  facilitate 
the  purchase,  monitoring  and  evaluation  of  computer-based  training 
systems  and  is  the  first  methodology  reviewed  here  directed  solely 
to  training  which  is  administered,  aided,  or  managed  by  computer.  It 
provides  a  standardized  structure  through  which  training  system 
costs  can  be  der ived  and  used  in  the  preparation  of  cost  and 
effectiveness  estimates. 

The  cost  methodology  is  focused  on  identifying  and  quantifying 
the  total  inputs  (men,  money,  material)  required  by  a  computer-based 
training  system  throughout  its  life-cycle.  Identified  inputs  are 
costed  in  terms  of  development,  procurement  and  operating  and 
maintenance  costs  to  attain  and  sustain  a  training  capability  over  a 
fixed  period  of  time. 


A  Training  Cost  Breakdown  Structure  is  used  to  specify  all  the 
major  cost  categories  of  resources  that  are  required  for  Computer 
Assisted  Instruction  ( CAI ) .  This  is  a  significant  addition  to  the 
work  reviewed  to  date  and  can  be  used  to  address  the  CAI  weaknesses 
noted  in  the  instructional  material  development  cost  categories  of 
both  TECEP  and  its  Litton  modification. 

This  is  not  a  weapon-specific  model  but  a  technology-specific 
model.  The  model  assumes  an  institutional  training  setting. 

For  purposes  of  this  methodology,  Seidel  and  Wagner  assume  the 
life-cycle  phases  to  be  sequential  and  of  a  pre-set  arbitrary 
length.  They  use  6  years  for  development,  one  year  for  procurement, 
and  up  to  8  years  for  operations  and  maintenance.  The  model  also 
assumes  instantaneous  rather  than  phased-in  fielding  of  the  system. 
For  training  effectiveness  measures  they  suggest  objectives-based 
achievement  or  time  measures  for  within-course  and  end-of -course 
criteria.  They  also  discuss  measures  such  as  attrition  rates, 
instructor  ratings  and  attitude  scales. 

While  effectiveness  criteria  do  reflect  training  requirements, 
this  specification  --  like  TECEP  --  does  not  establish  effectiveness 
priorities.  They  do  provide  guidance  to  the  trainer /decision-maker 
to  weight  effectiveness  measures  according  to  the  training 
requirements  that  drive  the  analysis. 

Development  phase  activities  include  applied  research, 
engineering  design,  analysis,  development,  test,  evaluation  and 
management  related  to  a  specific  computer-based  training  system. 
Included  in  the  procurement  phase  are  fabrication,  communication , 


reproduction,  packaging  and  shipping  and  other  activities  necessary 
to  transform  or  copy  the  tested  prototype  system  into  a  fully 
operational  system.  Operations  and  maintenance  activity  costs  would 
include  replacement  training  for  site  personnel. 

Seidel  and  Wagner  separate  the  work  effort  into  three  parts  by 
life-cycle,  suggesting  that  one  individual  be  responsible  for 
completing  the  specification  at  each  stage  and  also  suggesting  that 
for  the  entire  development  stage  and  part  of  the  procurement  stage, 
the  contractor  would  be  the  source  and/or  collector  of  the  needed 
information.  These  two  suggestions  may  present  problems. 

While  they  note  that  the  requirement  to  have  the  same  individual 
perform  the  data-gathering  and  analysis  at  each  phase  might  involve 
a  resource  needed  elsewhere,  they  insist  that  it  is  the  only 
approach  that  permits  data  collection  consistent  with  the  guidance 
and  formats  provided  in  this  specification.  The  reality  of  military 
rotation  presents  an  additional  drawback  to  this  requirement. 

Also,  contractor-supplied  information  in  the  early  development 
stage  appeared  such  a  problem  to  Matlick  et  al  in  their  1980  review 
that  they  recommended  that  explicit  contractual  requirements  be 
developed  to  assure  that  all  developmental  data  needed  for  CTEA 
analysts  became  available  in  timely  fashion. 

For  the  establishment  of  a  cost-effectiveness  ratio,  Seidel  and 
Wagner  begin  with  the  assignment  of  values  or  priorities  to  various 
dimensions  of  training  effectiveness.  These  authors  feel  that  a 
reasonable  cost-effectiveness  analysis  would  compare  the  best 
estimates  of  future  operations  and  maintenance  costs  a  id  not  include 


costs  that  were  expended  in  the  past.  They  suggest  that  the  ideal 
case  would  involve  comparing  the  cost-effectiveness  of  two 
alternative  training  systems  when  they  are  in  their  operations  and 
maintenance  phase,  and  strongly  warn  against  making  cost- 
ef  f  ectiveness  judgments  in  the  development  stage. 

They  do  however  provide  several  examples  of  cost-efficiency 
outputs  that  could  be  obtained  during  the  development  stage.  If 
training  requirements  have  been  established  and  graduation  signifies 
criteria  attainment,  they  suggest  two  ratios  as  indices  of 
efficiency:  (a)  projected  graduation  cost  per  graduate  either  for 
all  courses  supported  by  the  system  or  on  a  per-course  basis  and/or 
(b)  projected  hourly  cost  of  instruction  per  student  hours,  again 
either  for  all  courses  supported  by  the  system  or  on  a  per-course 
basis . 

Their  procedures  could  also  be  adapted  to  judge  the  efficiency 
of  a  computer-based  system  in  aiding  in  the  development  and  revision 
of  training  materials  independent  of  any  training  effectiveness 
judgment.  An  additional  benefit  would  be  the  capability  of  judging  a 
system's  efficiency  in  scheduling  training  resources  and  equipment 
and  keeping  records.  Seidel  and  Wagner  provide  an  important  caution 
against  confusing  another  efficiency  judgment  --  the  number  of 
students  completing  courses  --  with  training  quality. 

Their  approach  to  the  alternative  comparison  method  involves 
either  (a)  holding  effectiveness  constant  and  allowing  cost  to  vary 
or  (b)  holding  cost  constant  and  allowing  effectiveness  to  vary. 

They  caution  against  the  approach  recommended  in  the  TRADOC  TEA 
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handbook  of  allowing  both  cost  and  effectiveness  to  vary.  They  state 
that  this  may  lead  to  a  fallacious  attempt  to  minimize  cost  and  \ 

maximize  gain  at  the  same  time.  They  feel  strongly  that  infinite 
effectiveness  at  zero  cost  is  an  untenable  goal  which  should  not 
guide  the  cost-effectiveness  analysis. 

This  methodology  offers  additional  guidance  in  costing  computer- 
based  instruction,  underlining  the  importance  of  considering  all 
appropriate  costs  and  benefits  during  the  entire  operational  life  of 
a  system.  In  this  way,  "cost  avoidance"  factors  such  as  fewer 
instructional  personnel  or  facilities  or  less  training  time  --  which 
may  accrue  over  several  years  --  will  be  permitted  to  surface  and 
balance  the  large  initial  capital  investment  costs  of  implementing  a 
computer-based  training  system. 

The  model  makes  appropriate  use  of  economic  concepts  and 
includes  indirect  costs  and  inherited  assets.  Its  application 
assumes  formal,  school-based  training.  The  model  provides  extremely 
useful  guidance  for  including  all  the  relevant  costs  for  computer 
assisted  training.  It  is  unclear  if  the  model  has  been  validated. 

Computer-Based  Job  Skills  Education  Program  (JSEP) 

Another  development  in  computer-based  CTEA  was  reported  by 
Richard  Kraft  of  the  Center  for  Education  Technology  at  Florida 
State  University  and  Beatrice  Farr  of  ARI .  They  discussed  several 
key  cost  and  training  effectiveness  variables  that  would  be  used  to 
analyze  the  Job  Skills  Education  Program  for  ARI. 
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The  Job  Skills  Education  Program  (JSEP)  is  described  as 
"designed  to  provide  soldiers  with  job-related  basic  skills 
instruction  that  is  prerequisite  to  learning  their  skill  level  1 
and  2  job  tasks  during  their  first  duty  assignment.  Based  on 
extensive  job  analysis  of  the  94  largest  MOS  tasks  contained  in 
the  Soldier's  Manual  of  Common  Tasks,  JSEP  provides  functional 
basic  skills  instruction  on  MOS  specific  requirements."  (1983,p.l). 

It  appear  from  reading  this  "first  draft"  that  a  decision  has 
been  made  to  teach  these  tasks  with  computer-based  instruction,  that 
JSEP  itself  is  still  in  the  conceptual  stage,  and  that  the  decision 
involved  is  the  choice  of  the  right  configuration  of  personnel, 
terminals,  and  sites.  The  methodology  will  not  be  weapon  system- 
related. 

The  authors  were  planning  to  develop  a  model  which  will  address 
JSEP ' s  "unique"  requirements,  including  "the  self-paced  nature  of 
JSEP  (which)  rendered  assumptions  underlying  some  existing  CTEA 
models  inappropriate"  and  the  "open  entry,  open  access 
characteristics"  of  the  program  which  require  open  system  methods  as 
opposed  to  closed  system,  (p.vii)  The  authors  hope  to  have  a 
predictive  CTEA  model  developed  by  the  time  the  JSEP  system  has  been 
fully  defined. 

The  draft  describes  the  steps  in  this  development  as  defining 
operational  functional  constraints,  defining  appropriate  training 
configuration  possibilities,  and  quantifying  the  advantages  and 
disadvantages  of  each  design  alternative  for  individual  site- 
specific  requirements. 


The  proposal  anticipates  combining  research  and  development 
costs,  investment  costs,  operating  and  life-cycle  costs  with  the 
number  of  students  per  site,  the  student  flow  rate,  a  system  usage 
ratio,  the  rate  of  completion  of  assigned  lessons,  and  the  average 
number  of  hours  of  instruction  per  student  to  completion  to  produce 
costs  per  terminal  hour,  cost  per  instructional  hour,  class  hours 
per  training  center,  hourly  cost  per  graduate  and  direct  cost  per 
graduate . 

The  next  stage  of  JSEP  development  will  determine  whether  the 
cost  analysis  will  be  only  "real"  cost  oriented,  or  whether  it  might 
include  consideration  of  "sunk  costs  and  other  intangible  cost 
elements."  (p.3) 

The  proposed  development  steps  for  constructing  the  costing 
model  are  fairly  routine  but  they  do  include  a  step  to  describe  non- 
quantifiable  elements  of  training  cost  and  a  proposal  for 
"integrating  CTEA  with  a  decision-oriented  multi-criteria  utility 
analysis."  (p.4) 

The  draft  describes  an  ambitious  effort  which,  depending  on  data 
availability,  could  present  "cost  per  terminal  hour",  for  instance, 
as  a  function  of  the  cost  of  hardware  and  software,  courseware, 
design  and  development,  hours  of  usage  per  year,  number  of  terminals 
installed  relative  to  the  total  number  which  can  be  managed  by  the 
central  computer,  operations,  supervision,  and  maintenance,  (p.4) 

Even  more  ambitious  is  a  proposal  to  "investigate  the  range  of 
qualitative  effects  of  alternative  configurations  to  see  whether  any 
benefits  not  readily  expressible  in  dollars  could  perhaps  be 
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analyzed  using  rating  scales  and  evaluated  in  terms  of  'equivalent 
dollars',  for  example,  the  effect  on  projected  re-enlistment  rates 
by  a  configuration."  (p.4) 

As  indicated  by  the  authors,  this  is  a  first  draft  which  will 
undergo  several  revisions.  Attempts  to  obtain  an  updated  version 
were  unsuccessful.  There  are  additional  costing  objectives  briefly 
outlined  in  this  draft  which  need  further  clarification.  The 
proposed  CTEA  will  include  sensitivity  and  contingency  analysis  and 
costs  will  be  presented  in  present  value  form. 

In  developing  the  cost  model,  the  authors  list  several  models 
which  will  contribute  to  the  effort.  These  include  TECEP  and  the 
Cost-Effectiveness  Specif ications  developed  by  Seidel  and  Wagner 
reviewed  earlier.  Also  listed  is  an  Educational  Technology 
Assessment  Model  (ETAM),  developed  by  IBM  for  the  Navy  and  described 
as  a  follow-up  of  TECEP.  ETAM's  main  objective  is  described  as 
"quantifying  the  impact  of  educational  innovation  upon  existing 
courses."  (p.21) 

Another  cost  model  listed  in  the  proposal  is  one  developed  by 
Saisawan  Vadhanapanich  for  a  doctoral  dissertation  at  Florida  State 
University  in  1975.  Kraft  describes  this  cost  effectiveness  model 
as  "an  integration  of  the  separate  cost  and  effectiveness  models. 

Ths  purpose  of  the  integrated  model  is  to  combine  the  cost  per 
student  contact  hour  and  probability  of  system  effectiveness  into 
one  index  for  each  course  of  action  examined."  ( p . 2 1 ) . 

The  Vadhanapanich  model  described  in  the  dissertation  is 
designed  for  a  non-service  environment  and  does  not  account  for  the 
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costs  involved  for  a  military  student,  such  as  pay,  housing, 
benefits , etc .  It  appears  to  be  designed  for  a  civilian  education 
decision  maker  who  must  decide  in  what  year  to  purchase  computer- 
oriented  instructional  media.  The  proposal  does  use  discounting  and 
considers  residual  assets.  It  has  not  been  validated. 

As  noted  earlier,  the  JSEP-CTEA  proposal  is  in  first  draft  form 
and  revisions  were  anticipated.  ARI ' s  Technical  Director 
anticipated  that  when  all  phases  are  completed  "Army  decision  makers 
should  have  considerably  better  cost  and  training  effectiveness  data 
to  help  them  select  the  future  Army  CBI  system",  (foreword) 

In  their  review  of  cost  models  in  1980,  Matlick  et  al  nc _ed 
useful  elements  in  several  powerful  models  developed  for  the  Air 
Force  by  various  contractors.  These  efforts  were  not  incorporated 
into  their  developed  model  because  they  would  have  required 
substantial  revisions  in  Army  practice.  A  review  of  several  of 
these  contractor  developments  and  an  evolution  into  an  Army- 
compatible  methodology  follows. 


B-l  Systems  Approach  to  Training  (B-l  SAT) : 

Calspan  Corporation  developed  a  training  program  for  the  air 
crew  of  the  future  B-l  strategic  bomber  using  a  systems  approach  to 
training  in  1975.  Matlick  et  al  noted  that  Instructional  System 
Delivery  prescriptions  are  seldom  followed  as  rigorously  as  they 
were  in  this  case.  This  is  the  most  ambitious  attempt  reviewed  to 
assure  that  the  entire  training  system  is  considered  within  an 
orderly  and  complete  process. 


The  training  system  provides  for  transition,  recurring,  and 
upgrade  training  for  the  four  aircrew  positions.  The  instructional 
system  development  was  enhanced  by  the  concurrent  development  of  two 
computer  aids  .  These  involve  a  sorting  model  which  allows  efficient 
storage,  retrieval,  collating,  and  updating  of  mission/function/ 
task  analysis  and  supporting  data.  Behavioral  objectives  are 
arranged  hierarchically  and  examined  in  light  of  probable  entering 
student  qualifications.  Training  device  requirements  were  identified 
using  guidance  from  TECEP  modified  to  reflect  the  special 
requirements  of  the  B-l  SAT. 

A  major  goal  of  this  approach  was  to  increase  training  cost- 
effectiveness  by  eliminating  unnecessary  training  and  overly 
elaborate  training  devices  while  training  the  aircrew  to  the 
performance  criteria. 

The  second  computer  aid  is  called  the  Training  Resources 
Analytic  Model  (TRAM)  and  is  used  to  structure  and  schedule  courses, 
tracks  and  instructional  blocks.  The  program  examines  the  proposed 
training  system's  resources,  schedules  and  costs.  TRAM  is  exercised 
for  the  tradeoff  analyses  between  resources  and  instructional 
contexts.  Commonalities  among  objectives  might  suggest  their 
grouping  within  a  certain  instructional  context  but  this  may  be 
contradicted  by  the  cost  of  using  training  devices  and  facilities 
in  such  a  way. 

A  computerized  ability  to  perform  this  clustering  and  the 
subsequent  costing  would  be  a  significant  improvement  in  the  models 
thus  reviewed.  This  automated  approach  also  developed  different 
tracks  of  training  depending  on  expected  skill  and  knowledge  levels 


of  entering  students.  Another  output  from  TRAM  is  time-phased  life-cycle 
costs  and  resource  requirements  for  the  training  system. 

An  additional  significant  advantage  of  the  systems  approach  used 
for  the  B-l  SAT  is  the  provision  for  trade-off  and  sensitivity 
analysis  of  such  features  as  ratio  of  hours  spent  using  various 
training  media,  centralization  of  training  facilities,  and  training 
media  time  phasing. 

The  final  outputs  from  this  program  include:  (a)  a  description 
of  recommended  and  alternative  training  systems  (b)  a  syllabus  for 
each  course  and  (c)  description  of  required  media  and  facilities, 
costs  and  schedules. 

The  initial  source  of  information  for  the  effort  is  the  original 
task  analysis  data.  The  quality  of  this  data  base  establishes  the 
potential  quality  of  the  system  recommendations.  It  appears  from 
this  application  that  the  Air  Force  has  a  data  base  which  is 
sufficiently  developed  to  use  factors  for  such  items  as  research, 
development,  testing,  evaluation  and  other  preproduction  costs 
including  the  cost  of  the  first  unit  of  training.  Cost  estimates  for 
annual  operation  and  maintenance,  instructor  personnel,  trainee 
temporary  duty,  the  direct  costs  of  the  aircraft,  equipment 
operation,  maintenance  and  upgrade,  instructional  material  and 
facilities  are  all  calculated  using  factors  and  average  lead  times. 

It  would  be  a  significant  advantage  if  Army  cost  and  training 
analyses  efforts  could  lead  to  such  a  data  base. 

An  added  benefit  of  the  B-l  SAT  approach  was  the  assignment  of  a 
data  quality  designator  to  each  cost  element.  Cost  data  sources 
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included  manufacturers,  government  facilities  and  simulation  users. 

A  rating  system  indicated  predicted  accuracy  of  cost  data. 

The  cost  output  included  life-cycle  costs  and  used  the  constant 
dollar  base  year  approach.  The  methodology  is  training- 
prescriptive  and  cost-predictive  and  the  model  is  system- 
specific.  Jorgensen  (1979)  felt  the  real  value  of  the  work  had 
been  overlooked,  possibly  because  of  the  cancellation  of  the  B-l 
program.  (The  B-l  is  scheduled  to  be  replaced  by  the  Stealth 
plane.  Controversy  about  this  replacement  has  continued  into  the 
time  of  this  writing). 

Method  of  Designing  Instructional  Alternatives  (MODIA) 

The  Rand  Corporation  developed  a  methodology  for  the  Air  Force 
for  the  design  and  cost  analysis  of  an  instructional  system.  The 
results  were  reported  in  five  volumes  in  1977  including  an 
overview  of  MODIA,  options  for  course  design,  a  user  interface, 
a  resource  utilization  model  and  a  cost  model  called  MODCOM. 

MODIA  is  computerized  and  is  designed  specifically  for  formal 
training  in  the  five  Air  Training  Command  (ATC)  technical  schools. 
The  ATCs  account  for  the  bulk  of  Air  Force  technical  training.  The 
authors  note  that  over  one-third  of  the  300,000  different  course 
hours  in  the  technical  training  curriculum  are  substantially  revised 
or  newly  prepared  annually  and  provide  many  opportunities  for 
improvements  in  the  management  of  training  resources. 

MODIA  is  described  as  "a  systematic  process  for  planning  the  mix 


of  students,  instructors,  materials,  equipment,  facilities  and  the 


procedures  by  which  all  of  these  elements  work  together."  (Vol.4, 
Summary) .  It  can  be  used  to  design  new  courses  or  redesign  existing 
courses . 

The  "options  for  course  design  stage"  includes  guidelines  for 
design  choices  based  on  objectives,  student  population,  teaching 
policy  and  resource  constraints.  The  second  stage  involves  an 
interactive  question  and  answer  dialogue  between  user  and  computer 
which  expands  objectives  into  learning  events  and  develops  a  full 
course  description  including  teaching  strategy  for  learning  events. 
Charles  Jorgensen  of  ARI  (1979)  found  this  step  valuable  because  it 
allowed  the  users  to  see  the  impact  of  trade-off  decisions  they  have 
made  at  each  stage  of  the  process. 

The  results  from  this  second  stage  are  fed  into  the  Resource 
Utilization  Model  (RUM)  which  simulates  the  operation  of  the 
course  over  time  and  produces  detailed  reports  on  resource 
utilization  and  demand  as  well  as  student  flow  patterns.  The 
RUM  can  be  manipulated  to  reflect  the  inexact  nature  of  such 
events  as  arrivals,  student  ability  levels  and  other  parameters. 

The  RUM  simulations  are  then  fed  into  MODCOM,  the  cost  model, 
which  provides  total  course  costs  including  personnel  and  resource 
life-cycle  costs.  Some  of  the  requisite  cost  and  manning  factors  are 
stored  in  the  program  while  others  are  supplied  by  the  user. 

MODCOM  analyzes  manpower  and  other  training  expenses  for 
training  design  impacts.  The  manpower  categories  include  students, 
instructors,  administrative  personnel,  curriculum  personnel  and 
hardware  maintenance  personnel.  Other  cost  categories  are 


courseware  procurement,  hardware  procurement,  facility  construction, 
pay  and  allowances,  instructor  training  and  miscellaneous  operating 
expenses.  The  outputs  of  MODCOM  are  graduates  by  student  type; 
student  and  staff  man-years;  courseware,  hardware  and  facility 
characteristics;  total  course  costs  by  functional  element;  and  total 
course  costs  by  program  and  appropriations. 

MODIA  is  not  weapon-specific  and  the  authors  describe  it  as 
neutral  in  regard  to  the  effectiveness  of  training  design.  As 
Matlick  et  al  (1980a)  note,  the  original  input  to  MODIA  is  the 
product  of  subject  matter  experts  in  planning  and  developing  courses 
but  this  stage  is  not  specified  in  the  model.  They  further  note  that 
MODIA  could  be  used  to  compare  well-developed  training  program 
alternatives  but  that  its  required  level  of  detail  makes  it 
unsuitable  for  the  early  stage  of  the  life-cycle  of  a  weapon  system. 

The  cost  model  uses  a  five  year  time  horizon,  which  is  a  rough 
approximation  of  the  time  a  technical  training  center  course 
continues  before  being  completely  revised.  The  model  considers 
incremental  costs,  excludes  sunk  costs,  and  produces  discounted 
costs  and  benefits.  It  also  considers  inherited  resources  and 
residual  values.  Model  results  cannot  be  used  for  short-range 
budgeting  problems. 

MODIA  depends  on  the  existence  of  a  consolidated  data  base 
and,  as  noted  by  Matlick  et  al  (1980a),  it  would  require  substantial 
revision  to  be  suitable  for  Army  use.  Nevertheless,  there  are 
several  elements  in  its  cost  model  algorithm  which  should  be 
considered  in  the  model  to  be  developed.  These  include  (a)  the 


cost  per  instructor  of  initial  factory  training.  This  computes 
the  cost  of  providing  an  instructor  in  the  initial  cadre  with 
the  specialized  equipment  or  system  training  he  needs  for  developing 
and  conducting  a  course.  The  model  assumes  this  initial  specialized 
training  is  accomplished  at  a  private  contractor's  facility  with 
subsequent  training  of  instructors  provided  informally  by  the 
existing  instructor  nucleus  (b)  The  model  accounts  for  the  different 
personnel  costs  related  to  students  and  instructors  on  temporary 
duty  ( TDY )  or  permanent  change  of  station  (PCS).  The  latter 
would  consider  the  costs  of  transportation  of  personnel  and 
dependents,  and  shipment  and/or  storage  of  household  goods. 

If  it  is  not  possible  to  represent  computer  use  in  dollar  terms, 
MODI A ' s  authors  suggest  selecting  some  other  measure  of  resource 
utilization  such  as  Central  Processing  Unit  seconds  or  number  of 
core  bytes  used  and  keeping  track  of  that  utilization  outside  the 
cost  model. 

As  noted  earlier,  MODIA  would  not  be  helpful  in  the  early 
development  stage  of  a  weapon  system  and  would  require  substantial 
revision  for  Army  use.  Jorgensen  (1979)  feels  that  MODIA' s  value 
lies  more  in  the  area  of  program  management  than  in  actual  course 
development . 

Digital  Avionics  Information  System:  Training  Requi rements 
Analysis  Model  ( DAIS/TRAMOD ) 


Corporation  for  the  Air  Force  Human  Resources  Laboratory  and 
published  in  1978. 

Like  MODIA,  it  is  a  computerized  means  for  training  design  but 
it  makes  design  decisions  rather  than  simply  revealing  the  impact  of 
decisions  already  made.  The  model  is  used  to  decide  which  among  a 
group  of  tasks  should  be  trained,  and  how  --  given  a  series  of 
constraints  --  to  best  accomplish  a  well  defined  set  of  training 
objectives . 

TRAMOD  is  designed  to  operate  in  conjunction  with  a  reliability 
and  maintainability  model  and  a  system  cost  model  to  form  a  Life 
Cycle  Cost  Impact  Modeling  (LCCIM)  system  for  use  in  the  early 
stages  of  system  design.  The  LCCIM  system  was  developed  as  part 
of  and  was  to  be  applied  to  the  Digital  Avionics  Information 
System  advanced  development  program.  Each  model  within  the  LCCIM 
addresses  a  component  function  of  life-cycle  costing.  TRAMOD 
addresses  the  training  requirements  element  of  system  LCC. 

TRAMOD  operates  from  a  task  data  base  whose  preparation  is 
described  by  Matlick  et  al  as  a  manual  task  of  "considerable 
difficulty".  ( 1980a, III-18 )  The  task  input  includes  the  assignment  of 
values  to  a  number  of  parameters:  criticality,  learning  difficulty, 
frequency,  psychomotor  level,  cognitive  level,  and  estimated  time 
required  to  accomplish  training.  The  authors  suggest  that  "...the 
values  assigned  to  the  task  characteristic  parameters  should  be 
based  on  the  judgments  of  engineers  and  technicians  familiar  with 
the  equipment  upon  which  the  tasks  will  be  performed."  ( 1977, p. 12) 
Matlick  et  al  (1980a)  consider  this  a  major  change  in  the  way 


hardware  manufacturers  currently  perform  task  analysis  and  feel  this 
would  constrain  successful  use  of  the  method. 

Selecting  from  the  input  lists  the  tasks  to  be  trained,  the  user 
interacts  with  the  program  specifying  constraints  on  the  selection 
of  tasks  and  develops  blocks  of  tasks  to  be  trained.  Once  the 
training  blocks  have  been  determined,  TRAMOD  prints  a  series  of 
questions  to  which  the  user  must  respond  to  indicate  how  the 
training  times  and  costs  for  the  task  blocks  are  to  be  derived. 

Three  cost  options  are  available:  (a)  direct  input  of  training 
time  and  costs  for  each  task  block  in  the  input  data  set  (b) 
calculation  of  training  times  and  costs  with  user-selected 
regression  coefficients,  with  training  time  and  costs  being  linear 
combinations  of  the  task  characteristics  (c)  derivation  of  training 
time  and  costs  with  fixed  default  regression  coefficients  stored  in 
TRAMOD. 

The  program  uses  a  combination  of  optimizing  and  mapping 
techniques  and  results  in  specific  task  blocks  to  be  trained  with 
recommendations  on  training  mode  (school  or  on-the-job),  method  of 
instruction  and  media.  It  also  includes  a  time  estimate  and  a  cost 
estimate.  These  data  are  fed  into  a  training  program  generator  which 
schedules  resources  and  tasks  for  training. 

Designed  for  use  early  in  the  systems  acquisition  process  to 
facilitate  the  identification  and  timely  consideration  of  potential 
training  problems,  the  model's  primary  objective  is  to  enhance  Air 
Force  ability  to  avoid  potential  training  problem  situations  through 
action  within  the  design  process  itself. 
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TRAMOD  uses  comparable  operational  equipment  data  values  for 
these  early  development  estimates.  Course  length  and  cost  data 
information  for  the  technical  training  schools  was  available  but  on- 
the-job  training  times  and  costs  had  to  be  estimated. 

The  final  output  from  TRAMOD  includes  a  schedule  showing  the 
order  in  which  tasks  are  to  be  trained,  whether  a  task  is  to  be 
trained  in  OJT  or  school,  the  days  required  to  train  each  task,  the 
cost  of  training  each  task,  and  the  method  and  medium  for  each  task. 
The  model  provides  for  rerunning  program,  altering  decisions  and 
comparing  results. 

TRAMOD  considers  costs  but  not  effectiveness.  The  method  is 
cost  prescriptive  rather  than  predictive.  It  is  not  clear  that 
economic  concepts  such  as  discounting  and  residual  assets  are 
included  in  the  costing  estimates. 

Matlick  et  al  (1980a)  feel  it  would  be  valuable  for  comparing 
training  alternatives  resulting  from  various  constraints  such  as 
training  time  limits  or  equipment  shortages.  But,  like  MODIA,  the 
model  would  require  substantial  revision  for  Army  use. 

Coordination  of  Human  Resource  Technologies  ( CHRT ) : 

In  yet  another  Dynamics  Research  Corporation  effort,  Goclowski 
et  al  developed  Coordinated  Human  Resource  Technologies  for  the  Air 
Force  Human  Resource  Laboratory  in  1978  as  human  resource  assessment 
tools . 


This  ambitious  effort  combines  five  technologies  in  the  Weapon 
Systems  Acquisition  Process  (a)  maintenance  manpower  modeling  (b) 


instructional  systems  development  (c)job  guide  development  (d) 
system  ownership  costs  and  (e)  human  resources  in  design  tradeoffs. 

The  objectives  were  to  integrate  and  apply  the  technologies  to 
form  a  Coordinated  Human  Resource  Technology  (CHRT).  The  second 
objective  was  to  design  a  consolidated  data  base  to  support  CHRT 
application  by  establishing  a  common  source  of  information  for 
human  resource  technology  early  in  the  acquisition  process. 

A  Consolidated  Data  Base  (CDB)  is  established  for  each  weapon 
system  in  the  acquisition  process  and  it  grows  as  the  system 
develops.  The  five  technologies  draw  on  the  CDB  for  input  and 
many  of  their  outputs  go  back  into  CDB.  The  aim  is  to  have  one 
data  base  containing  information  on  reliability,  maintainability, 
maintenance  manpower,  operation  manpower,  training,  job  guides, 
and  system  ownership  cost.  It  is  designed  to  support  human 
resource  planning  during  the  acquisition  process  and  then  to 
support  operations  and  logistics  planning  after  system  deployment. 

The  Instructional  System  Development  (ISD)  in  CHRT  results  in 
training  concepts  during  the  concept  phase  of  a  weapon  system,  a 
training  plan  during  the  validation  phase,  and  a  fully-developed 
training  program  during  the  full-scale  development  phase.  Knerr 
et  al  (1984a)  note  that  the  ISD  phase  does  not  deal  with  features 
within  training  devices  and  does  not  predict  device  effectiveness 
or  transfer. 

The  Human  Resources  in  Design  Tradeoffs  phase  identifies  points 
in  system  development  where  the  selection  of  alternatives  has  a 
large  impact  on  human  resources. 


The  costing  portion  --  System  Ownership  Costs  --  was  not  fully 


developed  at  the  time  CHRT  was  published,  but  the  authors  were 
attempting  to  develop  a  model  to  identify  the  real  ownership  cost 
drivers.  They  suggested  it  might  be  necessary  to  tailor  SOC  for  a 
particular  system  during  the  conceptual  phase,  and  the  model  effort 
appears  to  be  aimed  at  operating  and  support  costs  rather  than 
acquisition  costs. 

At  the  time  of  publication,  the  System  Ownership  Cost  components 
were  to  include  (a)  support  investment  costs  which  included 
equipment,  job  guides  and  spares; (b)  operating  costs  which  included 
cost  of  aircrew  and  fuel;  and  (c)  support  cost,  which  included  cost 
of  depot  repairs,  inventory  management,  technical  record  data, 
equipment  maintenance  and  personnel  training.  These  costs  were  to  be 
described  in  constant  dollars. 

The  authors  were  planning  to  test  and  refine  the  methodology 
using  data  from  the  advanced  medium  Short  Takeoff  and  Landing 
( STOL ) transport .  At  the  time  of  their  1980  review,  Matlick  et  al  had 
not  been  able  to  identify  a  report  on  this  effort.  Like  MODIA  and 
TRAMOD,  this  Air  Force  development  would  require  a  major  conversion 
effort  for  Army  use. 

Military  MANpower  and  HARDware  Procurement  (HARDMAN) 

Dynamics  Research  Corporation  continued  its  efforts  to  develop 
human  resource  assessment  tools.  In  1980,  they  published  their 
HARDMAN  methodology  for  the  Navy.  This  is  another  effort  to 
determine  manpower,  personnel  and  training  requirements  during  the 
early  phases  of  the  weapon  system  acquisition  process  and  was  to 
involve  six  steps: 


(1)  Establish  a  Consolidated  Data  Base  by  developing  a  reference 
and  a  conceptual  system  and  evaluating  the  design  differences 
between  them.  This  step  also  involves  identifying,  collecting  and 
formatting  the  data  required  for  the  data  base. 

(2)  Determine  Manpower  Requirements  for  the  conceptual  system 
using  the  reference  system  as  a  point  of  departure.  Changes  in 
manpower  requirements  are  functions  of  design  differences  identified 
in  Step  1.  Knerr  et  al  (1984a)  note  that  this  step  assists  the  user 
in  providing  the  input  data  for  a  manpower  model  rather  than  being  a 
manpower  model  in  itself. 

(3)  Determine  Training  Resource  Requirements:  A  conceptual 
training  program  is  constructed  using  the  predecessor  and  reference 
training  program  to  reflect  system  design  differences.  The  impact 
of  task  and  course  changes  are  aggregated  to  determine  estimates  of 
training,  costs  and  resources.  Knerr  et  al  (1984a)  note  that  the 
task  action  verbs  are  analyzed  using  TECEP  task  categories.  They 
also  note  that  this  step  estimates  formal  school  training  costs  but 
not  on-the-job  training;  that  while  the  methodology  identified  the 
need  for  new  construction,  it  does  not  estimate  those  costs;  and 
that  it  makes  no  estimates  for  civilian  and  officer  training. 

(4)  Determine  Personnel  Requirements:  This  step  determines  the 
total  personnel  demand  of  the  reference  and  conceptual  system  which 
consists  of  personnel  required  "on  hand"  to  operate  and  maintain  the 
system  and  the  pipeline  personnel  who  must  be  "grown"  in  the  system 
to  consistently  meet  the  manpower  requirements  detailed  in  Step  2. 


(5)  Conduct  Impact  Analysis:  Determine  the  supply  of  those 
manpower  and  training  resources  required  by  the  conceptual  system 
and  measure  that  supply  projected  against  the  demand  determined  in 
Steps  2  through  4.  This  step  identifies  new  requirements  for 
skills,  training  and  training  resources,  design  and  other  sources  of 
high  human  resource  demand,  requirements  for  scarce  assets,  and  high 
cost  components  in  the  conceptual  system.  When  originally  developed, 
this  step  was  presented  in  outline  form  since  it  had  not  been  validated 

(6)  Perform  Tradeoff  Analysis:  This  step  prioritizes  the 
critical  requirements  according  to  their  impact  on  resources 
available.  A  range  of  potential  solutions  to  each  requirement  was 
to  be  iterated  to  develop  the  most  effective  response  to  each 
critical  resource  requirement.  Again,  at  the  time  of  original 
development,  this  was  presented  in  outline  form  without  validation. 

Reviewing  several  applications  of  the  methodology  in  1984,  Knerr 
et  al  note  that  personnel  who  apply  HARDMAN  typically  have  had 
military  experience,  have  engineering  backgrounds  and  have  performed 
task  analysis  in  the  past.  They  consider  the  process  which 
identifies  the  reference  system  somewhat  of  an  art  form. 

In  1981,  Dynamics  Research  Corporation  reported  on  the  results 
of  a  request  by  the  Army  Research  Institute  to  test  the  feasibility 
of  using  the  HARDMAN  methodology  (developed  for  the  Navy)  to 
determine  the  Manpower,  Personnel  and  Training  (MPT)  requirements 
of  an  emerging  Army  weapon  system.  They  reported  at  that  time 
that  the  methodology  had  already  benefitted  the  Navy  by  influencing 
the  choice  of  a  ship's  propulsion  system  based  on  adverse  MPT 
impacts  for  the  original  design. 


ARI  established  the  following  objectives:  (a)  determine  the 
ability  of  Army  data  to  support  the  methodology  (b)  determine  the 
utility  of  existing  HARDMAN  tools  for  Army  application  (c)  adapt 
when  necessary  both  the  data  and  analytical  tools  to  the  policy  and 
procedural  requirements  of  the  Army's  acquisition  process  (d) 
demonstrate  the  feasibility  of  the  methodology  for  Army  use  by 
applying  its  first  four  steps  to  a  major  Army  weapon  system. 

As  reported  by  Dynamics  Research  Corporation  (DRC),  the 
results  appear  to  support  a  general  conclusion  that  HARDMAN  is 
applicable  to  Army  systems.  They  note  that  while  data  of  sufficient 
quantity  and  quality  were  not  present  for  all  systems,  the  data 
that  were  present  were  very  good.  They  also  reported  on  an 
apparent  fragmentation  of  responsibility  in  the  early  phases  of 
the  Weapon  Systems  Acquisition  Process  which  led  to  the  lack  of 
coordination  and  delays  in  receipt  of  essential  data.  While 
noting  that  the  Army  recognized  the  problems,  DRC  pointed  out 
that  this  inhibited  a  more  extensive  front-end  analysis  for  this 
application. 

Like  other  developers,  DRC  recommended  the  establishment  of  new 
Army  data  bases  and  the  expansion,  integration  and  reorganization  of 
existing  data  bases  to  support  the  front-end  analysis. 

The  cost  of  personnel  which  must  be  supplied  for  the 
predecessor,  reference  and  conceptual  systems  were  derived  by 
multiplying  the  manpower  requirements  by  the  Composite  Standard 


Rates  for  Military  Personnel  Service  found  in  the  TRADOC  Resource 
Factor  Handbook.  The  bottom  line  value  for  each  system  reports 
on  annual  cost  to  support  the  manpower  requirements. 


Training  resources  and  costs  were  estimated  for  a  "steady  state 
year"  defined  as  the  first  year  in  which  an  Army  training  system  is 
producing  replacement  training  only,  that  is,  all  systems  have  been 
deployed  and  training  is  focused  on  filling  vacant  billets. 

Training  associated  with  the  Operational  Test  and  Evaluation  Tests 
of  the  proposed  system  and  training  associated  with  initial  fielding 
are  not  estimated  during  the  Training  Resource  Requirements  Analysis 
( TRRA )  step. 

TRRA  attempts  to  estimate  costs  and  resources  associated  with 
"first  order"  baseline  design  impacts  only.  That  is,  TRRA  will 
estimate  the  resources  and  costs  associated  with  training  the 
personnel  who  will  operate  and  maintain  the  system  or  subsystem.  A 
second  order  impact  --  such  as  requirements  for  training  of 
instructors  --  or  a  third  order  impact  --  such  as  the  requirements 
for  the  training  of  training  instructors  --  are  not  included.  This 
is  based  on  the  assumption  that  the  first  order  impacts  produce  the 
bulk  of  the  operational  and  support  costs  associated  with  the 
system. 

The  costs  produced  by  the  HARDMAN  method  were  individual  student 
cost  per  course,  average  individual  training  cost,  replacement 
personnel  training  cost,  cumulative  personnel  training  costs,  and 
instructor  costs. 


Since  this  initial  feasibility  test  for  ARI ,  the  HARDMAN 
methodology  has  been  transformed  into  Army-compatible  handbooks  and 
applied  to  several  Army  systems.  The  Army  adoption  of  the  HARDMAN 
technique  has  been  criticized  by  Hawley  (personal  interview,  1985) 
who  faults  HARDMAN’s  assumptions  that  current  operations  are 
effective  and  efficient  and  that  future  decisions  will  involve  more 
or  less  of  the  same. 

In  addition  to  this  lack  of  effectiveness  testing,  the  costing 
in  TRRA  ignores  what  may  be  significant  portions  of  training  costs 
in  the  early  development  and  fielding  phases  of  a  new  system,  and 
outputs  personnel  "replacement"  costs  only  for  a  fully-deployed 
weapon  system.  It  would  require  major  modification  to  provide  the 
information  needed  for  an  early  development  stage  CTEA. 

A  parallel  development  to  HARDMAN,  solely  Army-related,  has  been 
underway  since  1981.  A  review  of  the  evolution  of  this  methodology 
for  decision-making  follows. 

Training  Development  Decision  Support  System  (TDDSS): 

TRASANA  supported  the  development  of  an  automated  CTEA 
methodology  and  decision  support  system  by  Applied  Science 
Associates.  An  evolving  technology,  the  Training  Development 
Decision  Support  System  has  been  described  at  various  training 
conferences  since  1981.  It  has  been  applied  to  several  training 
situations.  Its  task  selection  procedures  are  based  upon 
Instructional  System  Development  guidelines  while  its  total  approach 
attempt  brings  it  close  to  the  output  discussed  in  the  B-l  SAT 


program . 


As  described  in  1981,  TDDSS  could  be  used  with  developing  as 
well  as  already  fielded  equipment  systems.  An  important  contribution 
of  TDDSS  is  its  thorough  human  factors  oriented  front-end  analysis 
of  the  job  and  its  tasks.  This  produces  decisions  involving  training 
requirements,  training  system  design,  device  specifications, 
training  management,  course  design,  system  design,  performance 
assessment  and  training  support  plans. 

While  the  primary  purpose  of  TDDSS  is  to  generate  data  for  cost 
and  training  effectiveness  judgments  for  pre-defined  performance 
objectives,  an  important  secondary  output  of  the  methodology  is  the 
provision  of  a  significant  portion  of  the  structure  and  content  for 
actual  course  design  --  the  proposed  Program  of  Instruction. 

TDDSS  provides  a  rational  objective  basis  for  selecting  tasks 
for  training  and  generates  information  for  designing  the  entire 
training  system  for  the  Military  Occupational  Specialty.  This  would 
include  maintenance  and  wartime  preparatory  training  requirements  in 
addition  to  the  resident,  unit  or  refresher  training  described  in 
previous  models. 

In  making  cost  and  training  judgments,  it  is  important  to 
identify  constraints  which  will  impact  on  design,  delivery  and 
management  of  a  course.  An  important  advantage  of  TDDSS  is  that  most 
of  these  constraints  are  identified  during  the  generation  of  a 
course  structure.  Constraints  can  include  time  and  resource 
availability,  instructor  number  and  qualifications,  instructional 
method,  philosophy  and  available  media  and  training  devices.  As 
noted  by  one  of  its  authors,  this  approach  has  the  advantage  of 
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allowing  for  the  consideration  of  such  factors  as  the  reliability 
and  maintainability  of  a  training  device  and  for  the  calculation  of 
the  impact  of  a  considerable  amount  of  downtime  for  a  device.  He 
noted,  for  example,  that  considering  this  variable  might  lead  to  a 
decision  to  select  several  part-task  trainers  rather  than  a  single 
full-task  device  whose  downtime  would  create  serious  bottlenecks  in 
the  training  program,  (personal  interview, 1985 ) 

After  a  general  course  structure  has  been  developed,  the 
next  step  in  TDDSS  is  to  specify  Training  Program  Alternatives 
( TPA )  to  meet  training  objectives.  While  the  authors  of  TDDSS 
describe  the  manipulation  of  such  variables  as  training  time 
(peacetime,  shortened  peacetime,  mobilization  time),  types  of 
simulators,  logistical  support,  student  characteristics  and 
numbers  to  develop  these  TPAs ,  they  warn  that  the  actual  selection 
of  training  alternatives  is  not  a  question  of  a  free  choice  of 
variables  or  levels  of  variables  but  is  importantly  influenced 
by  school  philosophies,  doctrine  and  local  standard  operating 
procedures . 

The  final  selection  in  TDDSS  establishes  "benefit- 
affordability",  first  dividing  the  alternatives  into  acceptable  and 
unacceptable  sets  based  on  effectiveness,  then  choosing  affordable 
sets  from  the  acceptable  sets.  Hawley  does  not  feel  a  variable  cost, 
variable  effectiveness  approach  is  viable. 

Elements  included  in  the  TDDSS  resource  costing  arc  similar  to 
several  previous  models  and  are  developed  for  each  alternative  using 
the  incremental  approach,  i.e.,  eliminating  all  common  costing 


elements . 
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The  choice  method  in  TDDSS  differs  from  previous  methods.  The 
authors  suggest  (a)  identifying  two  extreme  levels  of  a  few 
variables  that  would  have  the  greatest  impact  on  cost  and 
effectiveness  estimates  and  then  proposing  those  extreme 
combinations  as  the  alternatives  or  (b)  selecting  the  alternative 
most  acceptable  to  school  personnel  under  their  constrained 
conditions  and  one  or  more  ideal  alternatives  that  essentially 
ignore  the  constraints.  They  suggest  that  the  school  could  then  work 
toward  the  ideal  alternative. 

By  1983,  many  of  the  decision  processes  of  TDDSS  had  been 
computerized.  A  spreadsheet  was  described  which  would  allow  for 
rapid  cost  estimation  and  re-estimation  as  part  of  a  cost 
sensitivity  analysis.  Training  site  selection  (resident  or  on-the- 
job)  was  also  automated  by  this  time. 


The  TDDSS  approach  appears  the  most  promising  of  the  models 
reviewed  to  date.  Knerr  et  al  (1984a)  note  that  it  fills  a  need  for 
integration  of  device  prediction  and  prescription  models  with  models 
that  address  whole  training  programs.  While  not  totally  successful, 
reports  at  various  stages  of  the  model  development  indicated  that  an 
attempt  was  ongoing  to  build  a  consolidated  data  base  for  a  weapon 
system  which  would  be  a  considerable  improvement  for  Army  systems. 

TDDSS  responds  to  the  probability  that  there  will  be  analyst 
turnover  by  leaving  various  decision  audit  trails.  It  does  require 


highly- trained  learning  specialists  and  considerable  time  for  the 


front-end  analysis. 


The  production  of  a  program  of  instruction  and  course  which 
results  from  the  front-end  analysis  in  both  this  and  the  Bl-SAT 
approach  does  raise  one  technical  question.  It  produces  some 
instructional  development  cost  within  the  CTEA  itself.  It  is  not 
clear  if  this  could  or  should  be  separated  and  included  in  the 
system's  cost  and  how  it  would  then  be  evaluated. 


IV.  LESSONS  FROM  THE  LITERATURE 


Much  of  the  progress  in  CTEA  modeling  in  the  past  decade 
appears  to  be  concentrated  on  the  training  effectiveness  side  of 
the  issue.  The  measurement  of  cost  in  contemporary  models  is 
very  similar  to  the  framework  initially  proposed  by  Braby  et  al 
in  the  TECEP  model,  now  a  decade  old.  The  reader  is  struck  by 
the  evolution,  conceptually  and  methodologically,  during  this 
period  of  the  training  effectiveness  literature  in  comparison 
with  the  cost  literature.  This  may  be  a  reflection  of  the 
disciplinary  orientation  of  the  model  developers,  most  of  whom 
have  been  psychologists,  but  it  also  may  be  attributable  to  the 
absence  of  a  detailed  cost  accounting  system  supporting  the 
improvement  of  CTEA  cost  models. 

The  concepts  and  methods  of  cost  analyses  as  applied  to  CTEA 
are  well  established  in  the  literature.  The  weakness  of  this 
literature  seems  to  be  in  its  reluctance  to  explore  production 
and  process  function  approaches  to  obtain  more  useful  cost 
estimates.  (Solomon,  1985)  The  CTEA  cost  literature  reflects  an 
emphasis  on  the  specification  of  statistical  cost  functions 
where  the  cost  per  unit  of  output  is  described  as  a  function  of 
the  resources  employed  in  training.  The  formal  specification  of 
a  production  function  underlying  the  cost  function  is  rarely 


addressed.  As  a  consequence,  CTEA  cost  models  are  limited  in 
what  they  can  say  about  optimal  combinations  of  resources  for 
different  output  levels  of  training. 

To  address  this  deficiency,  Solomon  (1985)  suggests  an  expansion 
of  the  financial  accounting  system  data  base  to  provide  information 
on  parameters  such  as  resource  utilization  and  capacities  and 
scale  of  activities.  This  information  would  be  used  in  conjunction 
with  improved  cost  accounting  information  to  determine  optimum 
combinations  of  factors  for  different  levels  of  output. 
Alternatively,  activity  analysis  is  suggested  whereby  a  set  of 
related  activities  are  defined  in  a  training  process.  The  resulting 
process  function,  comprised  of  these  activities,  can  be  used  for 
attribution  and  evaluation  of  costs  in  the  existing  training 
process  and  may  provide  the  basis  for  estimating  and  evaluating 
costs  associated  with  changes  in  the  training  technology. 

Very  little  work  of  this  sort  has  been  undertaken.  Solomon 
refers,  for  example,  to  the  activity  analysis  undertaken  by  the  Army 
Training  and  Doctrine  Command  (TRADOC)  as  a  by-product  of  its  effort 
to  develop  statistical  estimating  equations  for  determining 
instructor  manpower  requirements  at  its  service  schools.  While  this 
suggests  a  new  direction  for  CTEA  cost  model  development,  it  does 
not  reject  the  statistical  cost  function  approach.  It  merely 
surrounds  this  approach  with  additional  information  linked  to  the 
training  production  function.  Insofar  as  the  statistical  cost 
function  approach  is  concerned,  there  are  several  important  lessons 
to  be  drawn  from  the  CTEA  cost  literature. 
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The  importance  of  specifying  model  objectives  is  stressed 
throughout  the  literature.  (Braby  et  al,  1975;  Matlick  et  al,  1980a) 
The  specification  of  the  cost  function  will  depend  on  whether  the 
objective  is  to  compare  alternative  training  technologies,  to 
determine  the  life-cycle  cost  of  a  system  or  technology  in  relation 
to  its  benefits,  or  merely  to  appraise  the  budgetary  impact  of 
fielding  a  new  weapon  system.  CTEA  cost  analysis  is  largely  focused 
on  the  first  of  these  objectives  with  an  eye  toward  cost 
minimization. 

Cost  Prediction  in  the  Conceptual  Stage  of  a  Weapon  System 

CTEA  studies  are  conducted  at  all  stages  of  the  weapon  system 
life-cycle,  but  the  most  important  in  terms  of  its  cost-saving 
implications  is  that  conducted  during  the  conceptual  stage  (DA 
Pamphlet  11-25  LCMM,  1975;  Jorgensen,  1979;  Matlick  et  al,  1980a). 

It  is  in  this  stage  that  the  best  potential  exists  for  low  cost 
design  modifications  of  the  system  to  enhance  effectiveness  and 
readiness.  Design  changes  at  development,  procurement,  or  operation 
and  maintenance  stages  in  response  to  CTEA  normally  entail  higher 
system  costs. 

This  theme  is  expressed  in  several  CTEA  models  which  were 


developed  with  this  objective  in  mind.  The  common  problem 
encountered  by  these  models  is  the  absence  of  historical  cost  data 
for  use  during  the  conceptual  stage  of  weapon  system  development.  In 


the  conceptual  stage,  the  source  of  cost  data  is  frequently  the 
contractor.  Some  have  suggested  that  the  development  of  a  financial 
accounting  data  base  for  CTEA  be  incorporated  into  contract 
requirements . 


Coordination  of  System  Design,  Training  Development,  and  Cost 
Analysis 

The  literature  suggests  that  predictive  CTEA  models  in  the 
conceptual  stage  benefit  from  the  close  coordination  of  those  who 
design  weapon  systems  and  training  devices  and  those  who  develop 
training  and  conduct  the  CTEA.  (Matlick  et  al  1980a;  TRASANA  TEA 
Handbook,  1980)  Coordination  among  these  individuals  leads  to  a 
better  understanding  of  data  needs  for  CTEA  and  well-established 
channels  of  communication  promote  the  discussion  of  system  design 
issues  and  the  early  detection  of  problems  leading  to  costly  system 
design  changes. 

Cost  Prediction  Through  Comparison-Based  Methods 

Comparison-based  methods  offer  one  approach  to  overcoming  data 
deficiencies  in  cost  estimation  during  the  conceptual  stage  of  a 
weapon  system  or  technology .( Klein , 1985 )  This  approach  calls  for 
the  identification  of  a  comparable  weapon  system,  training  device, 
or  task  to  that  under  development.  Subject  matter  experts  familiar 
with  the  comparable  system  or  technology  are  asked  to  use  their 
experience  in  projecting  the  cost  and  effectiveness  of  the  new 


system  or  technology.  This  methodology  uses  expert  judgment  to 
offset  the  data  deficiencies  of  cost  estimation  during  the 
conceptual  stage  of  development.  The  problem  addressed  by  this 
methodology  is  a  serious  one  warranting  further  investigation  of  the 
methodology's  potential  for  predictive  CTEA  modeling. 

The  Financial  Data  Base 

Improving  the  Army's  financial  data  base  for  CTEA  cost  analysis 
is  a  critical  need  as  seen  from  the  literature.  (Matlick  et  al, 
1980a;  Balcom  et  al,  1984)  This  issue  goes  beyond  the  problems  of 
cost  estimation  during  the  conceptual  stage  of  development  to 
include  later  stages  as  well.  Cost  data  maintained  for  budgetary 
purposes  do  not  always  conform  to  CTEA  needs,  since  the  latter 
exclude  resource  costs  that  are  common  to  alternative  training 
technologies  and  value  the  resources  used  at  their  opportunity 
cost.  The  development  of  a  CTEA  cost  model  must  keep  this 
constraint  in  mind.  A  proposed  cost  element  structure  by  Knapp 
and  Orlansky  (1983)  would  accommodate  data  for  management  and  cost- 
effectiveness  analysis  by  establishing  a  systematic  framework 
for  the  cataloging  and  separation  of  resource  costs. 

Cost  Estimation  for  Computer  Assisted  Instruction 

Computer  assisted  instruction  poses  some  special  problems  for 


cost  estimation,  primarily  in  courseware  development.  Orlansky 
(1985)  reports  that  courseware  development  represents  a  large  share 


of  system  cost  in  computer  assisted  instruction.  This  includes  the 
development  of  course  materials  for  use  with  computer-based  devices 
and  the  arranging  of  this  information  for  computer-based  control  in 
instruction.  Estimation  of  this  cost,  particularly  in  the 
conceptual  stage  of  system  development,  requires  special  expertise 
due  to  the  unique  nature  of  the  technology. 

rr'he  valuation  of  time-savings  using  computer  assisted 
instruction  also  requires  special  consideration.  The  introduction 
of  this  technology  may  result  in  systemic  reductions  in  training 
time.  Performance  standards  may  be  raised  and  the  time-savings  used 
to  meet  these  standards,  or  classes  may  simply  be  graduated  in  a 
shorter  time  period.  The  latter  would  result  in  cost  savings  which 
should  be  acknowledged  in  the  CTEA.  However,  if  the  technology 
results  in  varying  time  outcomes  for  different  students  with  no 
provision  for  using  the  time  saved,  such  as  moving  the  graduate  to 
the  next  training  stage  and  filling  the  training  slot,  then  the 
time-savings  should  not  be  considered  in  the  CTEA. 

Exposition  of  CTEA  Results 

The  literature  stresses  the  importance  of  properly  presenting 
the  CTEA  results  on  costs .( TRASANA  TEA  Handbook, 1980 ) .  Time  should 
be  spent  in  highlighting  the  important  cost  drivers  and  how  changes 
in  these  resources  influence  cost-effectiveness.  It  follows  that 


time  spent  by  the  analyst  in  estimation  should  be  focused  on  the 
important  cost  drivers.  Sensitivity  analysis  should  be  employed  in 


the  presentation  to  convey  the  robustness  of  CTEA  results  to  the 
analyst's  underlying  assumptions. 


Familiarity  with  Institutional  Constraints 


The  development  of  a  CTEA  model  and  its  use  requires  a  working 
familiarity  with  the  institutional  setting  in  which  training  occurs 
Specification  of  the  model  and  its  data  requirements  demand  this 
familiarity  as  does  the  application  of  the  model.  The  literature 
suggests  that  useful  applications  of  CTEA  will  consider  budgetary 
constraints,  existing  technologies  and  resources,  individual  and 
institutional  biases,  and  organizational  goals  and  objectives. 
(Braby  et  al,  1975;  Matlick  et  al,  1980a;  Hawley  et  al,  1983;  NATO 
Symposium,  1985.) 


Model  Specifications  Involvina  Trainina  Devices 


An  extensive  CTEA  literature  exists  on  training  devices.  T'  e 
cost  of  these  devices  will  be  driven  by  their  reliability, 
availability,  and  maintainability.  The  cost  model  specification 
should  anticipate  downtime,  parts,  spares,  and  repair  services. 
Cost  estimates  should  consider  a  contractor's  performance  history 
and  DOD  and  Army  supply  routines  and  schedules. 


Audit  Trails 

CTEA  analyst  turnover  can  be  a  serious  problem  to  maintaining 


continuity  and  consistency  in  model  development  and  application. 


(Matlick  et  al,  1980a).  It  can  greatly  complicate  efforts  to 
reexamine  earlier  decisions  in  applications.  As  a  consequence, 
several  cost  models  require  the  maintenance  of  an  audit  trail. 
(Hawley  et  al,  1983;  Klein, 1985).  A  paper  trail  of  decisions  is 
important  even  in  the  absence  of  analyst  turnover  as  one  attempts  to 
reconstruct  earlier  decisions. 

Generic  Models 

Several  model  developers  have  espoused  the  development  of  a 
generic  CTEA  model,  but  the  literature  suggests  that  this  goal  has 
not  yet  been  attained.  (Matlick  et  al, 1980a;  TRASANA  TEA 
Handbook, 1980 )  From  the  training  effectiveness  perspective,  the 
TDDSS  model  appears  promising,  but  little  specific  information  is 
available  presently  on  the  cost  framework  employed  in  this  model. 
Recent  model  development  efforts  have  focused  on  the  specification 
of  a  cost  element  structure  to  improve  the  relevance,  completeness, 
and  comparability  of  costs  among  alternative  training  technologies 
and  to  provide  a  common  basis  for  cost  data  generation  and 
collection.  These  efforts  are  also  intended  to  improve 
communication  among  the  various  organizations  concerned  with 
training.  (Knapp  and  Orlansky,  1983;  Seidel  and  Wagner,  1977) 

The  development  of  an  accepted  cost  element  structure  for 
defense  training  would  represent  an  important  step  toward  a  generic 
cost  model.  This  step  would  increase  the  likelihood  that  all 
resources  employed  in  training  would  be  enumerated.  Enumeration 


leads  to  the  valuation  of  these  resources  using  sound  economic 
concepts.  The  latter,  however,  requires  the  analyst's  judgment  and 
further  model  specification.  Determining  which  resources  are  fixed 
and  which  are  variable,  for  example,  is  critical  to  the  estimation 
process.  Generic  cost  models  can  at  best  specify  the  treatment  of 
broad  resource  classes. 

The  use  of  generic  models,  created  in  this  fashion,  will 
require  a  higher  degree  of  user  sophistication  than  more  specific 
models  tailored  to  particular  training  technologies.  The  latter 
permits  a  more  detailed  specif ication  of  the  model  with  a  resulting 
reduction  in  the  need  for  user  discretion. 
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